Apache Troubleshooting

Occasionally some requests to web servers cause memory/cpu problems – for the wrong reasons and sometimes the right reasons. I recently had a problem with a Google Search Appliance crawling a Drupal calendar but in hindsight it was probably my own fault as I had been playing with some of the settings below.

Which URLs are causing the problems?

If the Apache logs are any size (or split among vhosts) then tracking these down can be problematic. It is possible to add the process id to the apache log format using the %P parameter e.g.

LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” %P” combined

So if top is showing a big process it may be possible to track down the URLs served by the process. Remember to adjust your stats program to expect the %P. For awstats I needed:

LogFormat = “%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %other”

The scoreboard will also show the Urls currently being served by the apache processes.

Assuming mod_status is installed. Edit the httpd.conf and uncomment the following then restart httpd:

ExtendedStatus On

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 123.123.

Then check for processes/requests at http://mydomain.com/server-status


Scripts may run for no more than 100 seconds. Scripts running longer
than 100 seconds will be stopped automatically by the system/Apache.

RLimitCPU 100 100

Set Maximum of 25 processes at any one time

RLimitNPROC 25 25

Allow 10 MB to be used per-process

RLimitMEM 10000000 10000000

Entering ulimit -a will show the current system wide limits.

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16113
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16113
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

To amend a setting use:

ulimit -u 30

Which will limit ‘max user processes’ to 30. Be careful when setting these.
For permanent changes edit /etc/sysctl.conf to amend the kernel parameters.

Only useful for monitoring one vhost or log file. (much more info in this article)

apachetop -f /var/www/vhosts/mysite.com/statistics/logs/access_log

Other options:
-T 600    Will show details over a 10 minute period
Filter results with ‘f’ then ‘a’ then either ‘u’, ‘r’ or ‘h’

Kill Memory Hogs
If Apache is eating a lot of memory the following sledgehammer might buy some time. Run as a cron job every few minutes:

for i in `ps -efl | grep “httpd” |  awk ‘{  if ($10 > 16000) {print $4″ ” } }’`
  `kill -9 $i`

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>