Oct 052014
 

Investigation

The last few days I noticed my site wasn’t performing great and response times had gone up. When I had a change to look further, I could see that CPU usage was high and apache was running out of worker processes on a regular basis. However, overall site and bandwidth usage was low – there was no huge influx of visitors or bots pulling content which would explain the increased load. Looking deeper into apache’s access logs I quickly found the culprit(s):

The log above shows that during one second there were 7 posts to xmlrpc.php from 3 different IP addresses. These IPs are likely compromised or botnet servers which are attempting to brute force my server. XML-RPC functionality is used for extending certain functions in WordPress, like JetPack or the mobile client. However, it can be susceptible to brute force attacks on user accounts & passwords. I use very long (20+ characters) passwords which are random character strings, so the likelihood of anyone guessing any of the account passwords is extremely small, even with months and months of brute force attacks. However, the continued hammering on the server had introduced performance issues.

Resolution

While WordPress has been patched to prevent the xmlrpc.php attack for some time, compromised or botnet servers will still continue to hammer away on sites. The first order of business is to limit the functionality of XML-RPC to further reduce the surface area for the attack within WordPress itself. There are a number of ways to do this in WordPress, including disabling the functionality completely. In my case I want to retain some of the functionality in order to use JetPack and WordPress mobile. I can limit the functionality by using the iThemes Security and selecting Only Disable Trackbacks/Pingbacks:
XML-RPC

Additionally I wanted to block the three IPs which were hammering my site from even reaching WordPress in the first place. I could use .htaccess to prevent the IPs from being served WordPress & PHP, but I actually want to go one further and prevent them talking with apache at all in order to further limit CPU usage. I’m using IPTABLES in order to block these IPs at the firewall/network level, but there could be other ways of doing this depending on how your server is configured. Note that in the case of IPTABLES, the deny/drop commands should be above the allow commands in order for them to be processed properly:

The above is a simple fix. However, if you are seeing traffic on a number of different IPs and wanted to automate the firewall level block further, you could also implement a custom filter for xmlrps.php using Fail2Ban which http://xplus3.net/ has outlined well.

Results

CPU usage dropped dramatically as a result of limiting XML-RPC to specific functions and banning the 3 IPs which were hammering the server:
CPU usage

This has been a good reminder to review settings and logs on a regular basis.

Share it! Tweet about this on TwitterShare on FacebookEmail this to someoneShare on Google+Share on LinkedInPin on PinterestShare on Reddit