Bandwidth and avoiding the slashdot/digg effect

CS // 11 Apr 2009 // Comments Off on Bandwidth and avoiding the slashdot/digg effect

As I’ve spent the day doing some pre-emptive maintenance type stuff on our servers, I noticed that one of our servers appeared to have way slower ping times than normal from our other servers.

Taking a closer look I also saw that the logs said it having timeout problems talking to our database server.

A quick look at the server logs showed that the server in question was using 100% of its bandwidth for the last hour or so according to Monit.

A quick check with ntop (excellent web based network analysis statistics) and iftop (console based network traffic analysis) confirmed that 99.5% of the bandwidth was going to HTTP requests.
Strangely enough, the server wasn’t even stressed at all (I guess I’ve overspecced that one!)

I installed the apache2 module server-status ( a2enmod status, apache2ctl restart), and took a closer look at where the requests were going to.

In this case, all the requests were going to a single domain, to the detriment of other clients on the same server.

I decided to take a look at limiting the maximum bandwidth that that domain could use. I’ve used mod_bandwidth in the past for limiting apache virtual hosts, but sadly, it seems to have stopped development, so I took a look at whats currently available for apache2.

A few minutes googling later, I decided on installing mod-cband.

In debian, its fairly simple to install: (i’m assuming apache2 here)


apt-get install libapache2-mod-cband
apache2ctl restart


Now I had it installed, I had to configure it.
A quick read of the documentation shows that it needs some additions to the apache configuration.

I added this to my /etc/apache2/httpd.conf


CBandScoreFlushPeriod 1
CBandRandomPulse On


and then went to the virtual host in question, and added this:


CBandSpeed 1Mbps 10 200


The first number 1Mbps is how much total bandwidth the domain can use.
The second number is how many requests a second the domain can use.
The third number sets the maximum number of client connections to that domain.

I restarted apache2 – apache2ctl restart, and took a look at the excellent statistics provided by mod_cband

http://[theserverinquestion]/cband-status?refresh=15

…and then tweaked the CBandSpeed numbers based on what i was seeing (the values in the example above represent the optimum for the traffic i was seeing).

Any excess visitors over the (quite generously high) limits for that specific domain will see a 503 error, which can be customized to a page letting them know to come back later, as the server has exceeded its bandwidth limits.

The official documentation for cband is here – http://cband.linux.pl/, although their website doesn’t appear to be working currently (due to DNS timeouts).

A check of the logs showed that mod_cband was working nicely.

At around 8pm we started seeing heavy traffic (the far right), which started saturating the link about an hour later.
Once I’d installed mod_cband, and limited that domain, the traffic load dropped nicely.

Note that this server also acts as a backup server, so the increase in receive traffic at midnight is normal (the dip in the left side).

Conclusions:

Checking logs is a necessary evil 🙂

I also need to add some additional warning emails to the servers to notify me when things like this happen.
I already do this for some things like load, security scans, and failed logins, so its one more thing to monitor!

I’ve taken a look at this script here – http://sws.dett.de/mini/VHost_Traffic/ and will probably implement that at some point so I can see the biggest bandwidth usage in terms of client sites on a daily / weekly basis, with a view to moving them to a different server, and charging appropriately (yay, money!)

The good news is that it looks like the server in question can easily sustain slashdotting.