As part of our DNS migration in mid 2016 (June / July last year), we asked all our clients to update their DNS to the below servers.

ns1.digitalocean.com
ns2.digitalocean.com
ns3.digitalocean.com

Note that as of today (22nd Feb 2017), our previous DNS servers are now offline, and will no longer resolve.

If you are having issues with email and website hosting today (22nd Feb 2017), please ensure that your DNS has been updated as requested.

This will only affect clients that maintain their own DNS, if your services are managed by us, this was done last year.

If you are unsure on how to proceed, please contact us.

We are currently seeing some issues sending TLS encrypted mails to Outlook.com hosted email addresses.
This appears to only be affecting some of the Outlook.com hosted server ip addresses intermittently
213.199.154.87 / mail-db34087.inbound.protection.outlook.com
213.199.154.23 / mail-am14023.inbound.protection.outlook.com

If messages fail to be delivered, you will receive a bounce message similar to the following:

TLS connect failed: timed out; connected to 213.199.154.87.
I’m not going to try again; this message has been in the queue too long.

In the interim we have disabled TLS encryption to the affected addresses.
We are currently unsure if this is a Microsoft issue, or a China Firewall Issue, so this may or may not resolve the issue.

We will update this post when we have further information.

Update

Google has added another MX (mail server) for Google Hosted mail – alt4.gmail-smtp-in.l.google.com.

This does not currently appear to be blocked (unlike their other 4 MX servers), so we have removed the forwarding, and mail is transiting normally.

China has completely blocked gmail hosted mail as of today [28th April 2015]

This means that all mails heading to google’s servers is now blocked from Chinese ISP’s like ourselves.

Symptoms will include bounce messages where our server has given up retrying to send out the mail, as the remote server is not accessible over the Chinese internet.

EG –

Hi. This is the qmail-send program at mail.computersolutions.cn.
I’m afraid I wasn’t able to deliver your message to the following addresses.
This is a permanent error; I’ve given up. Sorry it didn’t work out.

:
Sorry, I wasn’t able to establish an SMTP connection. (#4.4.1)
I’m not going to try again; this message has been in the queue too long.

In the interim, we have added forwarding for all gmail addressed mail to transit through our oversea’s mail servers in the USA.

This should solve email delivery issues for gmail addresses – essentially anything addressed to someone @gmail.com

We are looking at solutions for resolving delivery to other google hosted mail clients, this will take some time to come up with a usable solution. In the interim, we can manually add routes on a server by server basis.

Be aware that this specific issue is out of our control, and we can only mitigate against it.

Examples of google hosted mail clients from recent queries/failure notices:

teamsequel.com – Their mail is served by google.

dig mx teamsequel.com

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> mx teamsequel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11757 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 2, ADDITIONAL: 0 ;; QUESTION SECTION: ;teamsequel.com. IN MX ;; ANSWER SECTION: teamsequel.com. 2320 IN MX 5 ALT1.ASPMX.L.GOOGLE.com. teamsequel.com. 2320 IN MX 5 ALT2.ASPMX.L.GOOGLE.com. teamsequel.com. 2320 IN MX 10 ALT3.ASPMX.L.GOOGLE.com. teamsequel.com. 2320 IN MX 10 ALT4.ASPMX.L.GOOGLE.com. teamsequel.com. 2320 IN MX 1 ASPMX.L.GOOGLE.com.

dreamonproductions.com – their mail is served by google.

dig mx dreamonproductions.com

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> mx dreamonproductions.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35828 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;dreamonproductions.com. IN MX ;; ANSWER SECTION: dreamonproductions.com. 3600 IN MX 5 alt1.aspmx.l.google.com. dreamonproductions.com. 3600 IN MX 1 aspmx.l.google.com. dreamonproductions.com. 3600 IN MX 10 aspmx2.googlemail.com. dreamonproductions.com. 3600 IN MX 5 alt2.aspmx.l.google.com. dreamonproductions.com. 3600 IN MX 10 aspmx3.googlemail.com.

Noticed that our incoming TLS connection queue was a little high – running at 60 concurrent connections for an hour or so.

A check of the queue revealed that all the connections were coming from a single IP – and were tying up the queue, making it a denial of service attack. This one ip address was connecting and reconnecting multiple times, hogging up all the connections.
Read more »

Seems that when it rains, it pours.

The gods were not content to give us only one issue today from an external provider, but two!

At approximately 7pm the network that includes our mail server was on got hit by a massive denial of service attack.
The nice people at Shanghai Telecom decided that they would simply shut off routing for the entire subnet as their optimal solution.

We have a nice graph of that happening here:

Note the sudden precipitous drop in network traffic starting at approximately 7pm, which lasted until approximately 8pm.

We also have images of the DoS attack [although not completely, as our network was null routed (shut off) for the brunt of the attack]

You can see the sudden increase in incoming traffic in this image below (which occurred before they killed the network completely).
The green line which indicates incoming packets suddenly goes sky high before the network people shut off the network.

Some of the other servers also got hit by this – notable our web servers, although they didn’t cut those off thankfully.
See below for a view of that traffic.

As the old curse goes – may you live in interesting times.
Some days are more interesting than others!

From 12:00 – 2:40pm today Shanghai Telecom was experiencing router problems for servers in the 61.129.88.xx address space in the data centre at WuSheng Lu (the main Shanghai Telecom building).
This affected 3 of our servers, and one of our clients managed servers.

Shanghai Telecoms official response below:

武胜机房托管了服务器61.129.88段在4月16日12:00-14:00出现无法访问连接，经检查该段均出现该种情况。我们公司技术向电信反映该情况后，经电信查看是由于该段中有主机发送广播包导致路由中毒（环路）而造成的，经过紧急的抢修最终恢复正常。此次给贵公司的日常运作带来很多不便，在此深表歉意。

Unfortunately once they had resolved their router issues, at around 3pm,  Shanghai Telecom decided to create some new ones, by arbitrarily rebooting all the servers in that address space.
Due to their actions, on reboot, our database server could not fully mount the data partition, and so a number of our client websites were unaccessible, as was our webmail service.

Repairing the damage caused by Shanghai Telecoms actions took around 2 1/2 hours.

Full services resumed at approximately 5:20pm

All services are currently running smoothly, although we do have some reports of connectivity issues from some clients.
If you are still unable to connect to the mail server, please turn off your ADSL modem or Router, and log onto the internet again.
(This will clear any route issues  in your router, and you should be able to connect successfully.)

Apologies for the inconvenience.

Lawrence.