Which URLs are causing the problems?
If the Apache logs are any size (or split among vhosts) then tracking these down can be problematic. It is possible to add the process id to the apache log format using the %P parameter e.g.
LogFormat “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” %P” combined
So if top is showing a big process it may be possible to track down the URLs served by the process. Remember to adjust your stats program to expect the %P. For awstats I needed:
LogFormat = “%host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot %other”
The scoreboard will also show the Urls currently being served by the apache processes.
mod_status
Assuming mod_status is installed. Edit the httpd.conf and uncomment the following then restart httpd:
ExtendedStatus On
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from 123.123.
</Location>
Then check for processes/requests at http://mydomain.com/server-status
RLimitMEM RLimitCPU
Scripts may run for no more than 100 seconds. Scripts running longer
than 100 seconds will be stopped automatically by the system/Apache.
RLimitCPU 100 100
Set Maximum of 25 processes at any one time
RLimitNPROC 25 25
Allow 10 MB to be used per-process
RLimitMEM 10000000 10000000
ulimit
Entering ulimit -a will show the current system wide limits.
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16113
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
max rt priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16113
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
To amend a setting use:
ulimit -u 30
Which will limit ‘max user processes’ to 30. Be careful when setting these.
For permanent changes edit /etc/sysctl.conf to amend the kernel parameters.
apachetop
Only useful for monitoring one vhost or log file. (much more info in this article)
apachetop -f /var/www/vhosts/mysite.com/statistics/logs/access_log
Other options:
-T 600 Will show details over a 10 minute period
Filter results with ‘f’ then ‘a’ then either ‘u’, ‘r’ or ‘h’
Kill Memory Hogs
If Apache is eating a lot of memory the following sledgehammer might buy some time. Run as a cron job every few minutes:
#!/bin/bash
for i in `ps -efl | grep “httpd” | awk ‘{ if ($10 > 16000) {print $4″ ” } }’`
do
`kill -9 $i`
done
