We are currently seeing some issues sending TLS encrypted mails to Outlook.com hosted email addresses.
This appears to only be affecting some of the Outlook.com hosted server ip addresses intermittently
213.199.154.87 / mail-db34087.inbound.protection.outlook.com
213.199.154.23 / mail-am14023.inbound.protection.outlook.com

If messages fail to be delivered, you will receive a bounce message similar to the following:

TLS connect failed: timed out; connected to 213.199.154.87.
I’m not going to try again; this message has been in the queue too long.

In the interim we have disabled TLS encryption to the affected addresses.
We are currently unsure if this is a Microsoft issue, or a China Firewall Issue, so this may or may not resolve the issue.

We will update this post when we have further information.

The SSL certificate for the all servers have been updated to use a wildcard certificate.

We *finally* changed over to use a wildcard cert, as pricing has come down enough to not warrant having separate certificates per server.
Our new wildcard certificate is valid until 2019.

What does this mean for you?

The bad news
Really old browsers won’t be able to open our site
If you are an XP user running IE6, you won’t be able to load our encrypted sites anymore. We strongly suggest you upgrade though if you fall into that category!
Same goes for those running Android 2.x (which is equally ancient in computer terms).

The good news
Email is now encrypted point to point using AES256 SHA encryption where possible, and webmail is SHA256 encrypted from server to your browser.
Mail servers that support it (i.e. all of ours, plus the major providers like Google, Yahoo etc, will send encrypted mail to our servers).
Mail Headers will include things like the below if encryption is supported –
Received: from usa4.computersolutions.cn (162.210.36.26) by mail.computersolutions.cn with AES256-SHA encrypted SMTP;

Lastly – our new cert gets us a test rating of A at the SSL Labs site.
https://www.ssllabs.com/ssltest/analyze.html?d=computersolutions.cn&latest

Some of our clients are experiencing delivery issues to some domains that use Gmail/Google for their email.

I previously covered that here – http://www.computersolutions.cn/blog/2015/04/gmail-and-other-google-hosted-mail-delivery-issues/

The issue is that China is still blocking Gmail/ Google hosted mail, and the recipient domain hasn’t setup their MX records correctly.

This is fine for servers outside of China, where all of googles mail servers (should) work, but breaks things for those inside China, where only a few servers are reachable.

Google hosted mail settings are here: https://support.google.com/a/answer/33915?hl=en

You’ll note that there are 5 different email servers that are listed in priority order.

Priority Mail Server
1 ASPMX.L.GOOGLE.COM.
5 ALT1.ASPMX.L.GOOGLE.COM.
5 ALT2.ASPMX.L.GOOGLE.COM.
10 ALT3.ASPMX.L.GOOGLE.COM.
10 ALT4.ASPMX.L.GOOGLE.COM.

For mail servers, the higher number is more important, so a priority of 1 will be the first server tried, then the next highest number, and so on.

If I try to connect to the servers from China.

telnet ASPMX.L.GOOGLE.COM 25
Trying 74.125.200.27…
(times out)

telnet ALT1.ASPMX.L.GOOGLE.COM 25
Trying 173.194.72.26…
(times out)

telnet ALT2.ASPMX.L.GOOGLE.COM 25
Trying 74.125.25.26…
(times out)

telnet ALT3.ASPMX.L.GOOGLE.COM 25
Trying 64.233.169.26…
Connected to ALT3.ASPMX.L.GOOGLE.COM.
Escape character is ‘^]’.
(yay, we have a winner!)

telnet ALT4.ASPMX.L.GOOGLE.COM 25
Trying 74.125.70.27…
Connected to ALT4.ASPMX.L.GOOGLE.COM.
Escape character is ‘^]’.
(yay, we have a winner!)

So, we can see that alt3, alt4 work, but none of the others do (as of 9th September 2015 from Shanghai)

So, some rudimentary testing shows that some servers work, and some do not.
How does that apply to real world examples.

Lets look at a non-working domain – ihg.com

dig mx ihg.com

;; ANSWER SECTION:
ihg.com. 600 IN MX 100 aspmx3.googlemail.com.
ihg.com. 600 IN MX 50 alt1.aspmx.l.google.com.
ihg.com. 600 IN MX 50 alt2.aspmx.l.google.com.
ihg.com. 600 IN MX 100 aspmx2.googlemail.com.
ihg.com. 600 IN MX 10 aspmx.l.google.com.

You should easily be able to see 2 things.
1 – that the MX records are not as per Google settings.
2 – that the 2 working MX records are not listed.

This means that while their MX records probably work oversea’s, they will not be deliverable from China. They need to amend their MX records to Googles recommended settings.

Lets look at another example.

dig mx rsms-west.com

;; ANSWER SECTION:
rsms-west.com. 6238 IN MX 30 alt2.aspmx.l.google.com.
rsms-west.com. 6238 IN MX 10 aspmx.l.google.com.
rsms-west.com. 6238 IN MX 40 aspmx2.googlemail.com.
rsms-west.com. 6238 IN MX 50 aspmx3.googlemail.com.
rsms-west.com. 6238 IN MX 20 alt1.aspmx.l.google.com.

Once again, we can see that the alt3, and alt4 servers are missing, and unfortunately none of the other listed servers are connectable from China.

Lastly, lets look at a working server

dig mx teamsequel.com

teamsequel.com. 12878 IN MX 1 ASPMX.L.GOOGLE.com.
teamsequel.com. 12878 IN MX 5 ALT1.ASPMX.L.GOOGLE.com.
teamsequel.com. 12878 IN MX 5 ALT2.ASPMX.L.GOOGLE.com.
teamsequel.com. 12878 IN MX 10 ALT3.ASPMX.L.GOOGLE.com.
teamsequel.com. 12878 IN MX 10 ALT4.ASPMX.L.GOOGLE.com.

You can see that they have the correct Gmail settings as per Gmail / Google settings page, and mail to them is deliverable (as alt3, alt4 are currently not being blocked by the beneficent government of China).

Unfortunately as this is an issue that is out of our control (MX records are incorrect, and China is being difficult), we cannot mitigate against it. The affected domains will need to amend their MX records appropriately as per the page here- https://support.google.com/a/answer/33915?hl=en.

Update

Google has added another MX (mail server) for Google Hosted mail – alt4.gmail-smtp-in.l.google.com.

This does not currently appear to be blocked (unlike their other 4 MX servers), so we have removed the forwarding, and mail is transiting normally.

China has completely blocked gmail hosted mail as of today [28th April 2015]

This means that all mails heading to google’s servers is now blocked from Chinese ISP’s like ourselves.

Symptoms will include bounce messages where our server has given up retrying to send out the mail, as the remote server is not accessible over the Chinese internet.

EG –

Hi. This is the qmail-send program at mail.computersolutions.cn.
I’m afraid I wasn’t able to deliver your message to the following addresses.
This is a permanent error; I’ve given up. Sorry it didn’t work out.

:
Sorry, I wasn’t able to establish an SMTP connection. (#4.4.1)
I’m not going to try again; this message has been in the queue too long.

In the interim, we have added forwarding for all gmail addressed mail to transit through our oversea’s mail servers in the USA.

This should solve email delivery issues for gmail addresses – essentially anything addressed to someone @gmail.com

We are looking at solutions for resolving delivery to other google hosted mail clients, this will take some time to come up with a usable solution. In the interim, we can manually add routes on a server by server basis.

Be aware that this specific issue is out of our control, and we can only mitigate against it.

Examples of google hosted mail clients from recent queries/failure notices:

teamsequel.com – Their mail is served by google.

dig mx teamsequel.com

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> mx teamsequel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11757 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 2, ADDITIONAL: 0 ;; QUESTION SECTION: ;teamsequel.com. IN MX ;; ANSWER SECTION: teamsequel.com. 2320 IN MX 5 ALT1.ASPMX.L.GOOGLE.com. teamsequel.com. 2320 IN MX 5 ALT2.ASPMX.L.GOOGLE.com. teamsequel.com. 2320 IN MX 10 ALT3.ASPMX.L.GOOGLE.com. teamsequel.com. 2320 IN MX 10 ALT4.ASPMX.L.GOOGLE.com. teamsequel.com. 2320 IN MX 1 ASPMX.L.GOOGLE.com.

dreamonproductions.com – their mail is served by google.

dig mx dreamonproductions.com

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> mx dreamonproductions.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35828 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;dreamonproductions.com. IN MX ;; ANSWER SECTION: dreamonproductions.com. 3600 IN MX 5 alt1.aspmx.l.google.com. dreamonproductions.com. 3600 IN MX 1 aspmx.l.google.com. dreamonproductions.com. 3600 IN MX 10 aspmx2.googlemail.com. dreamonproductions.com. 3600 IN MX 5 alt2.aspmx.l.google.com. dreamonproductions.com. 3600 IN MX 10 aspmx3.googlemail.com.

I’ve noticed a little spate of password attack attempts via Roundcube – a webmail program we use for mail over at https://mail.computersolutions.cn

Roundcube does have captcha plugins available which will mitigate this, but users will complain if they have to type in a captcha to login for mail.

Fail2ban provides an easy solution for this.

Roundcube stores its logs in a logs/errors file.

If I take a look at a sample login failure, it looks something like the example below

[09-Jun-2014 13:43:38 +0800]: IMAP Error: Login failed for admin from 105.236.42.200. Authentication failed. in rcube_imap.php on line 184 (POST /?_task=login&_action=login)

We should be able to use a regex like:
IMAP Error: Login failed for .* from

However fail2ban’s host regex then includes a trailing ., and fail2ban doesn’t recognise the ip.
I eventually came up with the overly complicated regex below, which seems to work:

IMAP Error: Login failed for .* from <HOST>(\. .* in .*?/rcube_imap\.php on line \d+ \(\S+ \S+\))?$

Lets add detection for that into fail2ban.
First up, we need to add roundcube into /etc/fail2ban/jail.conf

[roundcube]
enabled  = false
port     = http,https
filter   = roundcube
action   = iptables-multiport[name=roundcube, port="http,https"]
logpath  = [YOUR PATH TO ROUNDCUBE HERE]/logs/errors
maxretry = 5
findtime = 600
bantime = 3600

Note that we are not enabling the filter yet.

Change [YOUR PATH TO ROUNDCUBE HERE] in the above to your actual roundcube folder
eg /home/roundcube/public_html/logs/errors

Next, we need to create a filter.

Add /etc/fail2ban/filter.d/roundcube.conf

[Definition]
failregex = IMAP Error: Login failed for .* from <HOST>(\. .* in .*?/rcube_imap\.php on line \d+ \(\S+ \S+\))?$

ignoreregex =

Now we have the basics in place, we need to test out our filter.
For that, we use fail2ban-regex.
This accepts 2 (or more) arguments.

fail2ban-regex LOGFILE  FILTER 

For our purposes, we’ll pass it our logfile, and the filter we want to test with.

eg

fail2ban-regex  /home/roundcube/public_html/logs/errors /etc/fail2ban/filter.d/roundcube.conf  |more

If you’ve passed your log file, and it contains hits, you should see something like this:

Running tests
=============

Use regex file : /etc/fail2ban/filter.d/roundcube.conf
Use log file   : /home/www/webmail/public_html/logs/errors


Results
=======

Failregex
|- Regular expressions:
|  [1] IMAP Error: Login failed for .* from <HOST>(\. .* in .*?/rcube_imap\.php on line \d+ \(\S+ \S+\))?$
|
`- Number of matches:
   [1] 14310 match(es)

Ignoreregex
|- Regular expressions:
|
`- Number of matches:

Summary
=======

Addresses found:
[1]
    61.170.8.8 (Thu Dec 06 13:10:03 2012)
    ...[14309 more results in our logs!]

If you see hits, great, that means our regex worked, and you have some failed logins in the logs.
If you don’t get any results, check your log (use grep) and see if the log warning has changed. The regex I’ve posted works for roundcube 0.84

Once you’re happy, edit jail.conf, enable the plugin.
(set enabled = true), and restart fail2ban with

service fail2ban restart

As one of the main contention points people have with mail service is either the amount of spam they receive, or the amount of legitimate email we block, we’ve decided to put the solution in your hands.

We’ve added user access to the blocking implementation we use at Computer Solutions.

For a quite rerun on this our incoming mail rules are as follows:

• Sending Server has a valid Reverse DNS Entry
• Sending Server conforms to mail RFC’s
• Sending Server is not listed in any of the following Antispam Service Lists zen.spamhaus.org cblplus.anti-spam.org.cn cdl.anti-spam.org.cn bl.spamcop.net dnsbl.njabl.org
• Mail does not contain a virus, malware or similar content.
• Mail is addressed to a valid sender.
• Recipients mailbox is not full.

We’re giving you access to do what you want with regards to incoming spam blocks.
If you decide that our heinous blocking of senders who’s servers are _definitely_ listed in spam listings is not to your taste, then you can change that.

If you want to whitelist any incoming mail you can do the following:

1) Login as the postmaster account for your domain at http://rules.computersolutions.cn (in the example below, I’m editing my own account, you’ll need to use YOUR postmaster@yourdomain.com / password!)

2) Select Domain Wide Focus

3) Click Add a domain specific rule (this will apply to all messages received for your domain – i.e. anything @yourdomainname.com)

4) Setup appropriate rules (there are a number of options – in the example below I’m whitelisting all incoming mail).

5) Note that the System rules below are now greyed out (assuming you whitelisted as per example above).
Thats because they no longer apply!

In future we will be pushing clients to use this interface for their unblocking / blocking requirements, so that the needs of the few outvote the needs of the many, and your incoming email can go where no wo/man has gone before.

Lawrence.

As we’re a pro-active sort of ISP, we often take a look at ways to improve things for clients under the hood.

One of those improvements, was our recent DMARC implementation for our mail servers.

DMARC is a newish standard which adds to our existing SPF setup, by allowing us to publish methods that tell other providers how mail from us should be processed. It also allows us to receive reports from other providers as to how they’re processing our mail.

Read more »

While I usually am good at finding issues relatively quickly, I spent roughly 5 hours troubleshooting an issue today with incoming mail scanning.

What was the issue we were seeing?

Mail would randomly not get scanned by our mail scanner process (simscan), and simscan would exit with errors in various places.

eg

@4000000050d6d5601caac0b4 simscan: in run_ripmime
@4000000050d6d5601caac49c simscan: ripmime error
@4000000050d6d5601cab12bc simscan: exit error code: 71
@4000000050d6d5610478e3cc tcpserver: end 26607 status 0
@4000000050d6d5610478eb9c tcpserver: status: 5/150

What went wrong?

Initially I thought a recent update of our internal antivirus scanner software was to blame, as that was the only change.
I quickly eliminated that as an issue, by disabling the av test.
It worked for a few minutes, then started working incorrectly again.

My next thought was permissions, so I checked those against other servers, checked file permissions, checked ownership etc – all looked good.

Still no progress.

Eventually I recompiled most of the mail subsystem in case something funky was going on. Still no progress.

As it seemed to literally only happen to the simscan process, I decided to look into that code.

I compiled without rip mime initially, as I thought that was the issue, but again, it would work for one or two mails, then start breaking.

I decided to add in some additional debugging code inside simscan.c to see where things were breaking.

@4000000050d6d54c0d613584 simscan: in run_ripmime
@4000000050d6d54c0d613584 simscan: ripmime error
@4000000050d6d54c0d61a6cc simscan: exit error code: 71

I could see that it was calling the correct code segment, but still failing.
If I compiled without ripmime, it would work for a few minutes, then also fail on clamdscan.

I fiddled about with that for a good hour or two, until I decided to add more debugging, and recompile with ripmime again.

I added a few debug statements into simscan to let me know what was happening inside the ripmime function:

int run_ripmime()
{

/* fork ripmime */
switch(pid = vfork()) {
case -1:
if ( DebugFlag > 0 ) {
fprintf(stderr, "simscan:vfork ripmime error.\n");
}
return(-1);
...


I could see that simscan couldn’t fork ripmime.
What was weird though, was that if I changed to the simscan process, and ran the test manually, it would work.

Just not though qmail

After another hour or two of looking at incorrect things, I decided to go back and take a better look at the vfork issue.

Googling vfork fail linux eventually found my reason.

It ended up not being permissions related – vfork was actually failing, due to hitting its process cap.

qmail had reached its max limit of child processes, so simscan was getting called, then simscan would try to execute another process, and bam, max processes reached.

This was why it didn’t happen on the command line, but only in production.

The server is actually set to unlimited processes (see below), so this must probably have hit a linux kernel limit (unlimited doesn’t always mean unlimited!)

ulimit for server below:
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

ps -ef | grep qmail showed that we had a few hundred defunct qmail processes running, so did a

qmailctl stop

killall qmail-smtpd # which killed all the defunct processes)

qmailctl start

simscan started working once again. Next time it happens, I’ll be notified in the error log about vfork issues, and hopefully can spend some time to see why qmailctl restart doesn’t kill off the defunct qmail-smtpd processes…

This was quite a hard issue to debug, as all the issues and solutions online pointed to other common issues like permissions!

Eventually I’ll probably redo simscan to use fork() rather than vfork() as its not recommended.

Still, I learnt more useful things in the journey, so it wasn’t completely wasted time, although I wish it didn’t take me 5+ hours to debug!

Refs:
https://www.securecoding.cert.org/confluence/pages/viewpage.action?pageId=1703954

As our clients probably know, we run our own email servers.

Email is one of those things that looks simple on the surface, but isn’t necessarily so simple when it comes down to troubleshooting.

As is typical, the day started with a trouble ticket from a client letting us know that they were having difficulties receiving mail from one of their clients.

I was also given enough information to start troubleshooting without having to ask for additional information, which was nice, as this sometimes takes time to get.

We do have an email issues page with some notes and instructions for what is needed linked off of our webmail page; the issues page is here.

The bounce message supplied included headers, so it was fairly straightforward to work out what server they were using.

The sender (junglefish.com) was seeing the below as an error, when emailing us:

Failure notice from mxttb2.hichina.com:

I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.


As we have multiple mail servers in different countries, I first needed to identify which server they were connecting too, so I could check logs.

I usually do this by searching through our logs for occurrences of the senders domain.

If that doesn’t find any results, then I lookup what their mail servers should be and check for entries in our logs from those servers.

This doesn’t always guarantee a result, but in this case I had enough information to get a result.

Our sender was coming from junglefish.net, so I checked to see what mail servers they might use.

To check what mail servers are used, we check the MX (Mail Exchanger) records for their domain. In this instance I used dig – a dns lookup tool.

dig mx junglefish.net

junglefish.net. 3600 IN MX 10 mail.junglefish.net.
junglefish.net. 3600 IN MX 0 mail.junglefish.net.


We can see that mail is served by mail.junglefish.net for the junglefish.net domain. We can also see that the MX records are incorrect, as their are 2 entries pointing at the same server with different preferences. This won’t cause mail to be rejected though, so its not the issue we’re looking for.

Next we check the ip address for mail.junglefish.net

Some mail servers are on shared servers, so the domain name does not always match up with the ip address. We need the ip address as we need to search the logs for that, or addresses close to that range.

A simple ping should return their ip address.


ping mail.junglefish.net

Reply from 218.244.133.134...


In this instance, mail.junglefish.net points at 218.244.133.134

As mail servers also require valid reverse DNS entries, I also checked if that ip address has a valid reverse dns entry with nslookup.

nslookup 218.244.133.134
Server: ::1
Address: ::1#53

Non-authoritative answer:
134.133.244.218.in-addr.arpa name = out133-134.mxttb2.hichina.com.

In this case, their reverse DNS is out133-134.mxttb2.hichina.com
Just to double check thats valid, I pinged the name, and got a valid response.

ping out133-134.mxttb2.hichina.com
PING out133-134.mxttb2.hichina.com (218.244.133.134) 56(84) bytes of data.
64 bytes from out133-134.mxttb2.hichina.com (218.244.133.134): icmp_req=1 ttl=51 time=29.0 ms

So, we’ve done some basic checks –

We’ve checked that the sender domain has MX records.
We’ve checked that their mail server exists, and it has both forward and reverse dns.

Next up, I need to search the logs for connections from their server.

If I search for connections to our mail servers from 218.244.133 I saw the following:

USA – no results.
CN – no results.
CN2 – rejections.

Looking at the logs, I immediately saw that our server was rejecting their mail for failing MFCHECK

@400000004fb3951e1f2ee324 tcpserver: ok 8328 mail.smartshanghai.com:211.144.68.52:25 out133-134.mxttb2.hichina.com:218.244.133.134::51396
@400000004fb3951e28966374 jgreylist[8328]: 218.244.133.134: OK known
@400000004fb3952504477f44 qmail-smtpd[8328]: MFCHECK fail [218.244.133.134] junglefish.net

The MFCHECK code checks the domain portion of the envelope sender address (in the MAIL FROM command) to make sure it’s a real domain which has at least one MX record. This prevents spammers from being able to use phony domain names in their forged sender addresses.

In this case, our server was rejecting mail from them due to this failure.

As we didn’t have sufficient logs to see *why* this was happening, I turned on full connection logging for their ip address, and asked them to send us another mail.

They did so, and I checked the logs for what was happening.
(see below)

@400000004fb48fad0004630c tcpserver: ok 17315 mail.smartshanghai.com:211.144.68.52:25 out133-134.mxttb2.hichina.com:218.244.133.134::50611
400000004fb48fae061430c4 17315 > 220 mail.smartshanghai.com NO UCE ESMTP
400000004fb48fae090ae354 17315 < EHLO mxttb2.hichina.com 400000004fb48fae090decac 17315 > 250-mail.smartshanghai.com NO UCE
400000004fb48fae090df094 17315 > 250-SIZE 0
400000004fb48fae090df47c 17315 > 250-PIPELINING
400000004fb48fae090df864 17315 > 250 8BITMIME
400000004fb48fae0c26ec7c 17315 < MAIL FROM:
400000004fb48fc0396b0344 17315 > 451 DNS temporary failure (#4.3.0)
400000004fb48fc10a1b68f4 17315 < QUIT

In this case, their server started off the conversation with
EHLO mxttb2.hichina.com

As mail servers need to provide a valid domain that they claim to be sending from, I checked if mxttb2.hichina.com existed.
It didn't.

So, that was the reason.

Having identified that, I whitelisted their server ip address for the MFCHECK portion of our checks, and asked them to resend mail again.

Unfortunately this did not let them send mail as they failed another test!

@400000004fb495f92bd72ec4 qmail-smtpd[22644]: Received-SPF: error (mail.smartshanghai.com: error in processing during lookup of junglefish.net: DNS problem)

As SPF relies on TXT records, I tried to manually lookup their DNS

dig TXT junglefish.net

; <<>> DiG 9.7.3 <<>> TXT junglefish.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 11512 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

That failed.

Hmmmm.

I then tried using an oversea's server to lookup their domain.

dig ns junglefish.net

; <<>> DiG 9.7.3 <<>> ns junglefish.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41739 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;junglefish.net. IN NS ;; ANSWER SECTION: junglefish.net. 2931 IN NS ns54.domaincontrol.com. junglefish.net. 2931 IN NS ns53.domaincontrol.com.

That lookup worked.

So, we're getting somewhere.

We've identified a number of issues, and its starting to look like their DNS servers are blocked in China from a cursory test.

To confirm this I checked if using their DNS servers worked from within China.

Check from ns53 fails -
nslookup
> server ns53.domaincontrol.com
Default server: ns53.domaincontrol.com
Address: 216.69.185.27#53
Default server: ns53.domaincontrol.com
Address: 2607:f208:206::1b#53
> junglefish.net
;; connection timed out; no servers could be reached

Check from ns54 fails -
> server ns54.domaincontrol.com
Default server: ns54.domaincontrol.com
Address: 208.109.255.27#53
Default server: ns54.domaincontrol.com
Address: 2607:f208:302::1b#53
> junglefish.net
;; connection timed out; no servers could be reached
>

As those failed, I tried pinging their server.
ping ns54.domaincontrol.com
PING ns54.domaincontrol.com (208.109.255.27) 56(84) bytes of data.
^C
--- ns54.domaincontrol.com ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7056ms

Ping test fails

Next, I tried to see where it was failing with traceroute.

root@edm:/home/shanghaiguide# traceroute ns54.domaincontrol.com
traceroute to ns54.domaincontrol.com (208.109.255.27), 30 hops max, 60 byte packets
1 reserve.cableplus.com.cn (211.144.79.254) 5.723 ms 5.899 ms 6.078 ms
2 211.154.92.89 (211.154.92.89) 3.860 ms 3.878 ms 3.936 ms
3 211.154.64.94 (211.154.64.94) 3.545 ms 3.608 ms 3.686 ms
4 211.154.72.181 (211.154.72.181) 4.154 ms 4.421 ms 4.419 ms
5 202.96.222.73 (202.96.222.73) 4.657 ms 4.632 ms 4.635 ms
6 61.152.81.189 (61.152.81.189) 4.671 ms 5.401 ms 5.367 ms
7 61.152.86.50 (61.152.86.50) 6.387 ms 6.330 ms 6.301 ms
8 202.97.33.86 (202.97.33.86) 8.003 ms 7.941 ms 7.743 ms
9 202.97.33.254 (202.97.33.254) 7.829 ms 7.656 ms 7.647 ms
10 202.97.50.114 (202.97.50.114) 175.491 ms 175.982 ms 171.405 ms
11 202.97.52.218 (202.97.52.218) 155.507 ms 155.755 ms 155.750 ms
12 218.30.54.150 (218.30.54.150) 191.389 ms 191.306 ms 191.544 ms
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *

Traceroute fails (at 218.30.54.150)

whois 218.30.54.150
% [whois.apnic.net node-5]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html

inetnum: 218.30.32.0 - 218.30.55.255
netname: CHINANET-US-POP
descr: Chinanet POP in American
descr: 201 S. Lake Ave. Suite 604, Pasadena, CA 91101
country: CN
admin-c: CH93-AP
tech-c: CH93-AP
mnt-by: MAINT-CHINANET
changed: hostmaster@ns.chinanet.cn.net 20020221
status: ALLOCATED NON-PORTABLE
source: APNIC

So, it looks like there are 3 issues.

1) Invalid MX setup (although this is not going to stop mail from failing).
2) Their Mail server claims to be a non-existent domain.
(Note that this seems to have been subsequently rectified, as that domain now resolves).
3) Their DNS servers are blocked within China.

To solve that, I added an external DNS lookup from outside the country.

In summation, simple things may involve multiple issues, as we can see from above!

In this case, we resolved the issue by whitelisting the senders ip to bypass the MFCHECK failure. We then added external DNS resolvers to our Mail Servers to bypass the block.
Lastly informed the sender, and the client of all of the issues noted so that they could be resolved.

Occasionally even in a well maintained system, qmail has issues.

One semi-common issue I get to see, is when a server we send mail to doesn’t timeout. This ties up an outgoing mail slot. Over a period of time, this can lead to issues where the whole outgoing or incoming queue is sitting doing nothing, as every connection is tied up by ‘tarpitted’ connections.

Ideally Qmail should be able to cope with these.  There are settings in qmail to control how long a connection takes, and how long it should wait for.  These settings are covered in the following files (usually set in /var/qmail/control)

Read more »