Tale of a slow site

Posted by & filed under News, Troubleshooting.

We get asked a lot why choose WPOven over other cheap hosting say Godaddy , Hostgator etc or similar services Managed WP Hosts . Here is an example how we are different –

The Downtime

Recently a client from one of our other services approached us saying that his hostgator VPS is constantly very slow and hostgator support is constantly optimizing database , file system and adding plugins which is continually adding to site down time for days on end. When he approached at that moment also his site was quite slow since supposedly Hosgator was optimizing the database.  With constantly slow site he was in no mood to continue so we did not need much convincing more than “We will have you site up and working in hours” .

The Situation 

The client bought our professional plan and gave us the login details to the control panel of old site. Step one was to create backups and clone existing site at our end , this did not take long. The old site had all popular WP Caching plugins one can think of , they seem to have tried everything. We removed all the caching plugins and also recovered the database from an old backup since the current database was corrupted due to some so called optimizations.

After changing the name servers traffic started to show up on our server.  And within a 5 hours the sites were completely live.

Too Much Traffic ??? 

Once Name server migration was completed , client called us saying thanks but said though sites were working they were a little slow sometimes and should be fast as we had promised. We had enabled site wide varnish caching right now with no custom rules , so everything should have been good.  We checked the site and indeed it was not upto the mark, we logged into the server  and saw the bandwidth of the host was being used to the fullest. Since the alexa of the site was > 100k we instantly knew something was wrong,We compiled the varnish logs and saw that the server was receiving about 800k to a Million page hits per hour. For a site which was not so popular this was too high.We installed google analytics and checked live traffic and saw there were only about 130 – 150 users online at that time.  Running a few tools like iftop , netstat etc we saw a lot of valid http connections were being made to the site by random IP’s from all around the world even the site was completely German .  There was a big mismatch in the analytics report and what linux tools were reporting.

DDoS 

We notified our client that a small DDoS attack was taking place against him ,  he had no idea why it was happening and since when. Though our servers runs basic IPTables based protection they do not prevent valid http requests . We updated server firewall to  APF and installed DDoS Deflate and configured it to see if it would block IP’s which made too many connections. To our   dismay we discovered that this did not prevent anything , the connections which were coming did not do anything in excess and were like normal http requests remaining well below the configured number of connections and bandwidth usage. We even tried to rate limit using iptables , but that also did not do much. We tried Varnish based blocking but that did not do much and attacking connections were not much different from real connections.

The Solution 

The only thing consistent about these connections was they stayed there for a long time while a normal user connections would come and go. So we decided it would be best to write a script which ban IP’s using iptables which record a set number of connections over a period of time. So using iftop and a popular netstat command –

We started monitoring traffic and we were able find hosts with consistent connections for longer than usual.

A small script was developed on the server itself which ran every minute using cron and collected IPs which crossed a defined set threshold. This script kept filtering hosts which had connections for long time. After some optimizations of the script for detecting malicious IP’s we made it fully automated, the script started to ban most of the trouble some IP’s.

page_hits

Here is how the http traffic went down over the next few days , analytics using Custom Awstats parser built in.

page_visits

A obvious traffic increase was noticed once the site became faster.

Great, but …

After a few hours the site started to respond really fast thanks to varnish , but client got back to us that post views and likes were not registering. So next step was to configure varnish rules to allow certain parts of the pages so that post views took place and cache cleared when some certain actions took place. Our plugin Wpbase-cache is what we used as it was built for such setups.

Thank you so much

“Yes, i can see it now. Thank you so much. I´m very pleased with your service and will definitely recommend it and looking forward to a good working relations.” , The above comment described how the client felt. And it also made us happy about our efforts being appreciated. Though page views have gone down from millions to 100 – 200k / day as most of the , the number of unique visitors are growing. After all Everyone likes a fast site.

The Answer

Speed , Support and Security are the three things we promise and try our best to deliver on. Compared to other manged WP managed hosts like WP Engine , Pagely ,Zippykid etc ,  the user would have need a custom/enterprise plans with all of them which would be many times more expensive . And as for the cheaper hosts / VPS he was already with them and was not happy due to the support and constant downtime’s.

Leave a Reply