Tracking down High Server Loads

A list of useful commands that can be used to help identify a high server load:

Apache POST DOS Attach

• Check the amount of POST requests to Apache:

  
  /usr/sbin/tcpdump -A -nnn -s0 -l 'dst port 80' | grep -Eo "POST\ /.*"
  

• Find the culprit on a shared hosting environment:

  
  grep POST /var/www/vhosts/*/statistics/logs/access_log
  

• Find the IP address that is doing the most POST requests

  
  grep POST /var/www/vhosts/*/statistics/logs/access_log | awk '{print $1}' | sort | uniq -c | sort -n | tail -n 50
  

Finding a rogue PHP script using PHP's eval() function

find `pwd` -iname '*.php' -exec grep -H "eval(" {} \; > /tmp/eval_search.txt

Count the number of connections to Apache

netstat -an |grep ":80 " |wc -l

// View the connections to Apache.

netstat -an |grep ":80 "

// Count the connections to Apache per IP address

netstat -an |grep ":80 " | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | sort -n | uniq -c |sort -n

// Count the

netstat -tan| grep -v 'LISTEN'| awk '{print $5}'| grep -v 'and' |grep -v 'Address' |cut -d':' -f1 |sort -n | uniq -c | sort -rn | head -n10

// Sort IP addresses connected to Apache

netstat -tn 2>/dev/null | grep ':80 ' | awk '{print $5}' |sed -e 's/::ffff://' | cut -f1 -d: | sort | uniq -c | sort -rn | head

// Check for failed SSH logins

head -n1 /var/log/secure | awk '{ printf "Failed SSH Login Attempts Since: "$1" "$2": " }' && cat /var/log/secure | grep "Failed password" | wc -l && cat /var/log/secure | grep "Failed password" | perl -ne 'print "$&\n" while m#\d+\.\d+\.\d+\.\d+#g' | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 | uniq -c | awk 'length($1)>2'

IPTables

// Drop a single IP

/sbin/iptables -I INPUT -j DROP -s 192.243.55.132

// Drop a /24 range of IP addresses RedHat:

/sbin/iptables -I INPUT -j DROP -s 192.243.55.134/24

Ubuntu:

sudo iptables -I INPUT -j DROP -s 180.76.15.8/24

// Save the rules

/sbin/service iptables save

Removing a blocked IP address

/sbin/iptables -L INPUT -n --line-numbers | grep 'xxx.xxx.xxx.xxx'
/sbin/iptables -D INPUT X

// Good article explaining TOP, VMSTAT and other goodies. http://www.tummy.com/articles/isolating-heavy-load/