Fleaglass survives the 123-reg.co.uk VPS Disaster

As I’m sure your now aware fleaglass.com has been down since last Saturday (the 16th of April 2016).

The reason for this lies entirely with our, then, hosting company www.123-reg.co.uk

At 7am on Saturday morning someone at 123-reg.co.uk ran a script that had a catastrophic error in it. The result was that the script deleted the servers and websites of all their customers! This is a major embarrassment for a large web hosting group and judging by what we have seen on Twitter (https://twitter.com/123reg)  and facebook (https://www.facebook.com/123regfans/)  it has left some of their customers who had no backup or emergency plan with no website and potentially no business any more! – Fortunately, we were not one of those!

123-reg.co.uk have provided us with very little information and initially gave absolutely no information about what had occurred at their end. We were left wondering if this was a brief power outage or something more serious. They were providing us with updates at 6 hour intervals which equated to “we’re still working on it” and on Sunday evening said they would be working through the night to try and rectify the problem. It is now Wednesday the 20th of April  and still no solution.

We decided on Monday that it was too risky to wait any longer for 123-reg to sort out, if in deed they can, the problems so we decided to implement our “disaster recovery” plan.

Implementing this disaster recovery plan meant transferring everything from our own back up to a new server in another location with a new hosting company and then redirecting the web-traffic by changing the IP routing.  Implementing this was not something that we could easily switch on and off in an instant. It took a couple of days to build and set up new server and transfer the backed up files.

We regret that some of the data will be missing as our last system back up was 4 weeks ago. We are far more lucky that some who have lost everything as they didn’t back up their sites outside of 123-reg.co.uk

Although this is situation not our fault we are deeply sorry for all of the problems that this has caused.

Regards Matt & Keith
Team Fleaglass

 

You can read more about the unfolding catastrophe at

http://www.bbc.co.uk/news/technology-36072240

http://www.dailymail.co.uk/sciencetech/article-3545676/Sorry-deleted-internet-Error-domain-hosting-service-123-Reg-sees-hundreds-websites-going-offline.html

http://www.independent.co.uk/life-style/gadgets-and-tech/news/web-hosting-company-accidentally-deletes-part-of-the-internet-while-trying-to-clean-up-its-servers-a6989876.html

http://www.mirror.co.uk/tech/customers-furious-web-hosting-company-7780626