Seems that when it rains, it pours.

The gods were not content to give us only one issue today from an external provider, but two!

At approximately 7pm the network that includes our mail server was on got hit by a massive denial of service attack.
The nice people at Shanghai Telecom decided that they would simply shut off routing for the entire subnet as their optimal solution.

We have a nice graph of that happening here:

Note the sudden precipitous drop in network traffic starting at approximately 7pm, which lasted until approximately 8pm.

We also have images of the DoS attack [although not completely, as our network was null routed (shut off) for the brunt of the attack]

You can see the sudden increase in incoming traffic in this image below (which occurred before they killed the network completely).
The green line which indicates incoming packets suddenly goes sky high before the network people shut off the network.

Some of the other servers also got hit by this – notable our web servers, although they didn’t cut those off thankfully.
See below for a view of that traffic.

As the old curse goes – may you live in interesting times.
Some days are more interesting than others!

Early Monday morning it was bought to our attention that some clients could not receive mail, and others were having difficultly reaching their subdomains.

Investigation of the issue showed that the affected clients were missing DNS records.

Our DNS services are provided by a 3rd party – ServerBeach.
ServerBeach recently got bought out by a larger company Peer1.

The geniuses at Peer1 have managed to lose all our DNS info for over 400 of our domains during a migration from ServerBeach to Peer1 provided services, and now a substantial number of our clients have missing or incomplete DNS records.

They are aware of the issue, and are working on it.

We received an email notice about this this morning (after 7 or 8 hours of their DNS being offline)

Dear ServerBeach Customer,

At approximately 11:30am CST today we encountered an issue with our DNS services being imported into the new ServerBeach portal. This may be causing an interruption for certain domains that are hosted on the GeoDNS servers.

We are currently working diligently to correct this issue and will provide an update when DNS services are fully functional.

All of us at ServerBeach sincerely regret the inconveniences associated with this incident, and will fully stand behind our Service Level Agreement. We apologize for this disruption and thank you for your continued patronage and understanding.

If you have any questions regarding this issue, please open an online ticket athttps://my.serverbeach.com/ or contact our support team at 1-800-741-9939.

Regards,

Brian Daffern
Director of Support

We have raised a number of tickets with ServerBeach / Peer1 regarding this, and hope for a speedy resolution to the matter.

As an interim measure I’ve added forwarding records for domains affected, so that mail received on the webservers that host affected domains know to forward mail to the correct mail server.

ServerBeach is updating a forum post regarding this issue here:
http://forums.serverbeach.com/showthread.php?t=7919
(Note that in the second post they incorrectly say it has been resolved. It has not been resolved yet)

If ServerBeach / Peer1 cannot resolve this by this evening, we may have to look at changing to a different DNS provider.
This is a last resort measure though, as DNS server changes take 1-2 days to propagate throughout the internet.

Apologies for any inconvenience, and we hope you can bear with us while we resolve the situation.

If you have any questions, please address them to our support email: support at computersolutions.cn