7
Another outage!
Seems that when it rains, it pours.
The gods were not content to give us only one issue today from an external provider, but two!
At approximately 7pm the network that includes our mail server was on got hit by a massive denial of service attack.
The nice people at Shanghai Telecom decided that they would simply shut off routing for the entire subnet as their optimal solution.
We have a nice graph of that happening here:
Note the sudden precipitous drop in network traffic starting at approximately 7pm, which lasted until approximately 8pm.
We also have images of the DoS attack [although not completely, as our network was null routed (shut off) for the brunt of the attack]
You can see the sudden increase in incoming traffic in this image below (which occurred before they killed the network completely).
The green line which indicates incoming packets suddenly goes sky high before the network people shut off the network.
Some of the other servers also got hit by this – notable our web servers, although they didn’t cut those off thankfully.
See below for a view of that traffic.
As the old curse goes – may you live in interesting times.
Some days are more interesting than others!
Early Monday morning it was bought to our attention that some clients could not receive mail, and others were having difficultly reaching their subdomains.
Investigation of the issue showed that the affected clients were missing DNS records.
Our DNS services are provided by a 3rd party – ServerBeach.
ServerBeach recently got bought out by a larger company Peer1.
The geniuses at Peer1 have managed to lose all our DNS info for over 400 of our domains during a migration from ServerBeach to Peer1 provided services, and now a substantial number of our clients have missing or incomplete DNS records.
They are aware of the issue, and are working on it.
We received an email notice about this this morning (after 7 or 8 hours of their DNS being offline)
Dear ServerBeach Customer,
At approximately 11:30am CST today we encountered an issue with our DNS services being imported into the new ServerBeach portal. This may be causing an interruption for certain domains that are hosted on the GeoDNS servers.
We are currently working diligently to correct this issue and will provide an update when DNS services are fully functional.
All of us at ServerBeach sincerely regret the inconveniences associated with this incident, and will fully stand behind our Service Level Agreement. We apologize for this disruption and thank you for your continued patronage and understanding.
If you have any questions regarding this issue, please open an online ticket athttps://my.serverbeach.com/ or contact our support team at 1-800-741-9939.
Regards,
Brian Daffern
Director of Support
We have raised a number of tickets with ServerBeach / Peer1 regarding this, and hope for a speedy resolution to the matter.
As an interim measure I’ve added forwarding records for domains affected, so that mail received on the webservers that host affected domains know to forward mail to the correct mail server.
ServerBeach is updating a forum post regarding this issue here:
http://forums.serverbeach.com/showthread.php?t=7919
(Note that in the second post they incorrectly say it has been resolved. It has not been resolved yet)
If ServerBeach / Peer1 cannot resolve this by this evening, we may have to look at changing to a different DNS provider.
This is a last resort measure though, as DNS server changes take 1-2 days to propagate throughout the internet.
Apologies for any inconvenience, and we hope you can bear with us while we resolve the situation.
If you have any questions, please address them to our support email: support at computersolutions.cn
Archives
- November 2024
- November 2019
- October 2019
- August 2019
- April 2019
- February 2017
- September 2016
- June 2016
- May 2016
- September 2015
- August 2015
- June 2015
- April 2015
- December 2014
- October 2014
- September 2014
- July 2014
- June 2014
- April 2014
- October 2013
- July 2013
- May 2013
- April 2013
- March 2013
- January 2013
- December 2012
- October 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012
- March 2012
- December 2011
- November 2011
- October 2011
- September 2011
- July 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- May 2009
- April 2009
- March 2009
- February 2009
- January 2009
- December 2008
- November 2008
- October 2008
- September 2008
Categories
- Apple
- Arcade Machines
- Badges
- BMW
- China Related
- Cool Hunting
- Exploits
- Firmware
- Food
- General Talk
- government
- IP Cam
- iPhone
- Lasers
- legislation
- MODx
- MySQL
- notice
- qmail
- requirements
- Reviews
- Service Issues
- Tao Bao
- Technical Mumbo Jumbo
- Things that will get me censored
- Travel
- Uncategorized
- Useful Info