System Outage Issues Resolved (April 16th)

CS // 16 Apr 2009 // Comments Off on System Outage Issues Resolved (April 16th)

From 12:00 – 2:40pm today Shanghai Telecom was experiencing router problems for servers in the 61.129.88.xx address space in the data centre at WuSheng Lu (the main Shanghai Telecom building).
This affected 3 of our servers, and one of our clients managed servers.

Shanghai Telecoms official response below:

武胜机房托管了服务器61.129.88段在4月16日12:00-14:00出现无法访问连接，经检查该段均出现该种情况。我们公司技术向电信反映该情况后，经电信查看是由于该段中有主机发送广播包导致路由中毒（环路）而造成的，经过紧急的抢修最终恢复正常。此次给贵公司的日常运作带来很多不便，在此深表歉意。

Unfortunately once they had resolved their router issues, at around 3pm,  Shanghai Telecom decided to create some new ones, by arbitrarily rebooting all the servers in that address space.
Due to their actions, on reboot, our database server could not fully mount the data partition, and so a number of our client websites were unaccessible, as was our webmail service.

Repairing the damage caused by Shanghai Telecoms actions took around 2 1/2 hours.

Full services resumed at approximately 5:20pm

All services are currently running smoothly, although we do have some reports of connectivity issues from some clients.
If you are still unable to connect to the mail server, please turn off your ADSL modem or Router, and log onto the internet again.
(This will clear any route issues  in your router, and you should be able to connect successfully.)

Apologies for the inconvenience.

Lawrence.