Seems that when it rains, it pours.

The gods were not content to give us only one issue today from an external provider, but two!

At approximately 7pm the network that includes our mail server was on got hit by a massive denial of service attack.
The nice people at Shanghai Telecom decided that they would simply shut off routing for the entire subnet as their optimal solution.

We have a nice graph of that happening here:

Note the sudden precipitous drop in network traffic starting at approximately 7pm, which lasted until approximately 8pm.

We also have images of the DoS attack [although not completely, as our network was null routed (shut off) for the brunt of the attack]

You can see the sudden increase in incoming traffic in this image below (which occurred before they killed the network completely).
The green line which indicates incoming packets suddenly goes sky high before the network people shut off the network.

Some of the other servers also got hit by this – notable our web servers, although they didn’t cut those off thankfully.
See below for a view of that traffic.

As the old curse goes – may you live in interesting times.
Some days are more interesting than others!

Early Monday morning it was bought to our attention that some clients could not receive mail, and others were having difficultly reaching their subdomains.

Investigation of the issue showed that the affected clients were missing DNS records.

Our DNS services are provided by a 3rd party – ServerBeach.
ServerBeach recently got bought out by a larger company Peer1.

The geniuses at Peer1 have managed to lose all our DNS info for over 400 of our domains during a migration from ServerBeach to Peer1 provided services, and now a substantial number of our clients have missing or incomplete DNS records.

They are aware of the issue, and are working on it.

We received an email notice about this this morning (after 7 or 8 hours of their DNS being offline)

Dear ServerBeach Customer,

At approximately 11:30am CST today we encountered an issue with our DNS services being imported into the new ServerBeach portal. This may be causing an interruption for certain domains that are hosted on the GeoDNS servers.

We are currently working diligently to correct this issue and will provide an update when DNS services are fully functional.

All of us at ServerBeach sincerely regret the inconveniences associated with this incident, and will fully stand behind our Service Level Agreement. We apologize for this disruption and thank you for your continued patronage and understanding.

If you have any questions regarding this issue, please open an online ticket athttps://my.serverbeach.com/ or contact our support team at 1-800-741-9939.

Regards,

Brian Daffern
Director of Support

We have raised a number of tickets with ServerBeach / Peer1 regarding this, and hope for a speedy resolution to the matter.

As an interim measure I’ve added forwarding records for domains affected, so that mail received on the webservers that host affected domains know to forward mail to the correct mail server.

ServerBeach is updating a forum post regarding this issue here:
http://forums.serverbeach.com/showthread.php?t=7919
(Note that in the second post they incorrectly say it has been resolved. It has not been resolved yet)

If ServerBeach / Peer1 cannot resolve this by this evening, we may have to look at changing to a different DNS provider.
This is a last resort measure though, as DNS server changes take 1-2 days to propagate throughout the internet.

Apologies for any inconvenience, and we hope you can bear with us while we resolve the situation.

If you have any questions, please address them to our support email: support at computersolutions.cn

Setting up OpenVPN was a real PIA for a number of reasons DNS, crap documentation, and general issues with vpn clients.

My working notes are below:

Install OPENVPN from tar.gz or apt-get install…
Generate key’s etc (tons of other tutorials on that)

Prelim info
My vpn server has a static ip address, in the 66.xx range. Our local client machines use a 192.x range (typically).
I setup a tun address for 10.1.0.1 for the server.

As we don’t want to have routing issues, I set openvpn to use the 10.x range for any vpn connections.
(Essentially all clients connected to the openvpn ip will get a 10.1.0.x address).

I also force clients to use our DNS server (more on that later), as China does some DNS lookup interceptions which break stuff if you are using a tunnel. I also don’t use openvpn on the standard port 1194, as I was seeing mysterious tcp resets when using the common vpn ports. Amazing how that happens in China. Lastly, I’ve put in on port 8080 for our users, as this seems to work without issue.

To do all that, I created an openvpn.conf file with the following:

port 8080
#proto tcp
#dev tun
proto udp
dev tap
ca keys/ca.crt
cert keys/server.crt
key keys/server.key
dh keys/dh1024.pem
server 10.1.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
comp-lzo
user nobody
group users
persist-key
persist-tun
status /var/log/openvpn-status.log
verb 3
client-to-client
push "redirect-gateway"
push "dhcp-option DNS 10.1.0.1"
link-mtu 1456
mssfix 1412
cipher AES-256-CBC

(You can read the standard install stuff for your own key generation)

Next we need to tell our server to route stuff appropriately for vpn traffic

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -A INPUT -p udp --dport 8080 -j ACCEPT
iptables -A INPUT -i tun+ -j ACCEPT
iptables -A FORWARD -i tun+ -j ACCEPT
iptables -A INPUT -i tap+ -j ACCEPT
iptables -A FORWARD -i tap+ -j ACCEPT
iptables -t nat -A POSTROUTING -s 10.1.0.0/24 -o eth0 -j MASQUERADE

(You’ll need to change the 10.1.0.0 to your actual vpn user subnet if you change in the openvpn.conf)

OpenVPN should start, and be connectable.

My client config looks approximately something like this:

client
dev tap
proto udp
remote mysupersekritvpnserver.com 8080
comp-lzo
verb 3
mute 20
nobind
persist-key
persist-tun
cipher AES-256-CBC
ca ca.crt
cert my.crt
key my.key

my.crt, my.key, ca.crt should be copied / generated from the server, and copied over to the client machine.
mysupersekritvpnserver.com should be changed to your server name.
We use Mac’s mostly, so we use tunnelblick, copy that config in, check the “Set NameServer” box in Details.

You should be able to connect now and ping remotely with that.
Next, we need to setup DNS

For the longest time I couldn’t get this working, despite me reading and re-reading the doc’s.

We use dnscache for dns lookups on our servers. DNS Cache allegedly allows lookups from other ip addresses by sticking whats allowed into /etc/dnscache/root/ip

This wasn’t working at all.

Eventually I twigged that dnscache binds to one ip address, and ignores the others, which is why local lookups worked, but tunnel started ones didn’t.

Took me a while to see that though. Was only when I did an nmap 10.1.0.1 and saw port 53 was closed, that I realised, despite the misleading fscking documentation which says “just add the ip address for the computers allowed to connect” to the dnscachefolder/root/ip, you really need to bind it to all the ports you will want lookups to work for.

Which is not clearly mentioned in any documentation I saw on the net.

I ended up making another dnscache specifically for our tun address on 10.1.0.1, and telling it to allow queries from the actual server ip 66.x, and from 10.x, *then* it started working.

Hours of fun and joy.

Worth it though, I can now connect to bookface and toobyou, yay!

Unfortunately, its back to the techie stuff for a few posts!

Here are my crib notes on installing NGinx on one of our client servers.

Add spawn-fcgi (cos its split from litettpd now) http://redmine.lighttpd.net/projects/spawn-fcgi/wiki/SVN


cd /downloads
svn co svn://svn.lighttpd.net/spawn-fcgi/trunk spawn-fcgi


No svn..grrr


apt-get install subversion subversion-tools
svn co svn://svn.lighttpd.net/spawn-fcgi/trunk spawn-fcgi
cd spawn-fcgi
./autogen.sh
./configure
make
make install


spawn-fcgi should be happily installed in /usr/local/bin/spawn-fcgi now

make sure we have php5-cgi
apt-get install php5-cgi

Check spawn-fcgi runs –
In my case not, had to rebuild eAccelerator @#$@#$!, did that, and all ok


/bin/spawn-fcgi -f /usr/bin/php5-cgi -a 127.0.0.1 -p 53217 -P /var/run/fastcgi-php.pid


Note port 53217 can be any unused port from some high unused number though to 65535
We’re going to be installing NGinx from debian packages, but probably better from source long term…

apt-get install nginx

pico /etc/nginx/nginx.conf


change some default settings


user www-data;
worker_processes 1;

Now we add our virtual hosts in /etc/nginx/sites-available


pico /etc/nginx/sites-available

Lastly, we add our fast-cgi settings, using our port from above 53217


fastcgi stuff here


If anyone wants to mess around with the settings for the Huawei eHome router EchoLife HG522-c (typically the ones supplied with the “3M or 4M” connection), then here are the user / pass settings.

Site: http://192.168.1.1/
User: telecomadmin
Pass：nE7jA%5m

Useful if you want to rejig the QoS settings.

If that login doesn’t work, try this – which is usually seen on the HG226 models –
Site: http://192.168.1.1
User: fiberhomehg2x0
Pass: hg2x0

The other standard modem HG520S is easier – admin / admin

Might be useful for some folks. I’m mostly posting here for myself, as I’ll probably forget and need to google it later.

One of the (not so) fun things about China is that almost everything needs to be licenced.
As part of the China bureaucracy plan, all forms of wheeled transport in Shanghai require a licence (yes, even bicycles!).

Riding motorized transport without one is not recommended, as this can lead to fines, deportation, and jail in worst case scenario’s.
This has been documented enough times by those unfortunate enough to knowingly break the law.

Electric Bikes/ Scooters are not exempt from requiring a licence, despite what the nice man at the shop selling you the bike, or others might say.

To be street legal in Shanghai, you need a plate.

1) Only bikes purchased in Shanghai can get a Shanghai plate.
Keep your official receipt (fapiao) when you buy the bike, as it needs to be used to get your licence.

2) As of a law passed in April 2008, Shanghai Electric bikes have to be:

* Under 40kg.
* Not capable of speeds faster than 20km/hr.
Effectively this means only bikes < 36v. * Listed in the allowed vehicle database for Shanghai. Legal bikes have a 15 digit unicode (like a car VIN) which is unique. Manufacturers have to apply for a production licence for this, and not all have done so. A list of licenced manufacturers is available here: http://www.shbicycle.com/Article/ShowArticle.asp?ArticleID=6230

If you own a bike that is older than that date that does not conform to the above, and you had a legal licence at that time, then it can be renewed each year, and its still legal.

3) To licence an electric bike/scooter/moped you need to visit the police station for your district.

You will need to bring:

* – The fapiao for the bike
* – Valid form of ID (Passport for foreigners)
* – The bike
* – Money to pay for the licence (11rmb currently)

—–

FAQ’s:

Can I use a legal plate from another province?
eg Jiangsu…
Possibly No, with some caveats –
It is not legal to ride with an out of town plate unless it is a weekend, or public holiday. You can be fined 200RMB if caught riding outside of these times, although this is unlikely. This is a grey area though, so police attitude to this may vary/change. While having a plate is better than not having a plate, it may not help if you have an accident.

Can I carry a passenger?
According to the law, no.

Is is illegal to drive without a plate?
Yes. Being a foreigner does not exclude you from following the law.

Do I need a driving licence for an electric bike/moped/scooter?
No.

Is there an official list of legal bikes?
There are 2 official sites where you can check if a bike is legal – http://www.shbicycle.com and http:/www.shjtaq.com

A current list of legal bikes with photo’s is here:
http://www.shbicycle.com/Photo/ShowClass.asp?ClassID=84

You will need Chinese reading abilities to read either site.

Instructions below for non illiterate:

市民在购买电动自行车时，可以通过上海市自行车行业协会网站上（HTTP://WWW.SHBICYCLE.COM），或上海交通安全信息网（网址： HTTP://WWW.SHJTAQ.COM，点击首页右上方的 “车/牌/证公告”栏目中的“上海电动自行车—上牌备案登记表”）查询该目录。

Are the cool Vespa lookalikes legal?
No.
The Vespa lookalikes / Spicy Motors bikes are not street legal for various reasons – weight, voltage, speed (unless you have a 2008 model licenced prior to 8/2008).
Additionally, they are not listed in the police database of allowed models.

…but XXX / Bike shop / my friend etc says its legal?
In short: They’re lying.

Caveats:
If you are in living a city other than Shanghai this may be ok.
Different cities, different rules.
If the bike is 2nd hand, and has an existing legal Shanghai licence, and it has been renewed each year, then it is also ok (albeit extremely unlikely).
Otherwise, see the short answer.

What about repair / warranty?
By Law, bikes have a “三包” (threefold warranty).

If you have issues within 7 days the seller is legally obligated to give your money back if you so wish.
Within 2 weeks, you are guaranteed a replacement bike.
Within 1 year, if you have more than 2 of the same type of failure, they have to replace, or you can get your money back less depreciation.
Consumables such as lights are not covered by this, nor are collisions..

Parts have the following warranties (by law):
Motor: 2 years
Frame, Fork, Charger, Controller, Battery: 1 year

*For batteries, failure is deemed as holding < 60% of original charge. More details here: http://www.shbicycle.com/news/ShowArticle.asp?ArticleID=43

Note that it is while it is illegal for shops to sell bikes that cannot be licensed, many still do.

Where can I licence my electric [bike/scooter/moped]?

Addresses for each district:

Bao Shan
宝山区 凇兴西路长征新村23号 56672872

Chang Ning District:
长宁区
天山路11弄12号
62747031

No 12, Lane 11, Tian Shan Lu

Hong Kou:
上海市丰镇路118号/上海市水电路1656号
上午8：30—11：30；下午13：30—17：00 周五下午不
受理 65161561

Feng Zhen lu / 1656 Shui Dian Lu
Closed Friday morning.

Huang Pu District:
黄浦区
山东南路49号
63289464

49 South Shan Dong rd

Jing An
静安区
昌平路372号
62539361

372 Chang Ping road (off Shaanxi road)

Lu Wan District:
思南路、香山路 交界拐角处－－卢湾区非机动车管理处
卢湾区 思南路46号 63275000
46 Si Nan lu / Xiang Shan lu

Min Hang District:
上海市沪闵路4888号（莘庄镇靠近颛桥）
上午8：00—11：00；下午13：00—16：30
6489 1010-3015

4888 Hu Min lu

Nan Hui District:
南汇非机动车管理所：

上海市南汇川南奉公路6116号
上午8：30—11：30；下午13：00—17：00
电话58021896

Nan Shi District (Southern parts of City?):
南市区 中山南一路161弄5号 63138859

Pu Dong District:
浦东非机动车管理所：
上海市浦东新区杨高中路1500号上午：9：00—11：30；下午13：30—16：30 周五下午不受理电话28946594
or
浦东新区 浦东南路3640号 58394097

1500 Yang Gao Middle Road
or
3640 Pudong South Road

Putuo District:
普陀区
芦定路325号 52811677

Xu Hui District:
龙吴路2388号，徐浦大桥下面
2388/2138 Long Wu Lu, underneath Xu Pu bridge

徐汇区 龙吴路2138号（徐浦大桥） 64340579

Yang Pu District:
杨浦区
双阳路357号
65433020

Zhabei:
闸北非机所
上海市天目中路707号
上午8：30—11：30；下午13：30—17：00 周五下午不
受理 63172110

Also 闸北区 共和新路1985号 56650065

707 Tian Mu Middle Road

[Update 23/Oct/09: Hotmail has fixed this issue now]

Our logs were showing lots of repeated send failures from Hotmail.
A closer investigation of the issue has revealed that Hotmail has suddenly decided that the mail RFC’s are too good for them to follow.

RFC’s are the standards which define how things work. When people don’t follow the standards, this makes things break.

In this case, it meant that all mail from Hotmail was being rejected, this is a Hotmail is broken issue!
Getting Hotmail to change their broken setup is likely to be non-productive – there are already a few pages of complaints about it on their site, complete with the boilerplate totally useless replies from drones who don’t understand the issue, despite it being helpfully spelled out for them.

See here –
http://windowslivehelp.com/community/p/127432/474962.aspx
http://windowslivehelp.com/community/t/123986.aspx

Unfortunately, while bouncing invalid email content is correct from a technical perspective, our clients need to be able to receive mail from Hotmail.
As an interim solution, I’ve patched qmail to allow for bare linefeeds.

This was fairly easy – a small patch to qmail-smtpd.c, a recompile, then restart qmail-smtpd.

To patch, look for switch(state) in qmail-smtpd.c, and remove the straynewline(); calls, so that barelinefeeds are accepted.
Code to change below:

case 0:
if (ch == '\n') { state = 1; break; }
if (ch == '\r') { state = 4; continue; }
break;
case 1: /* \r\n */
if (ch == '.') { state = 2; continue; }
if (ch == '\r') { state = 4; continue; }
if (ch != '\n') state = 0;
break;
case 2: /* \r\n + . */
if (ch == '\n') return;
if (ch == '\r') { state = 3; continue; }
state = 0;
break;


As the Wiki for fail2ban is a little less than explanatory than it could be (and they reversed my edits which made the instructions clearer), here are my own notes on setting up fail2ban to block pop3 attacks.

Have been seeing sample dictionary attacks on some servers for a while now from random ip addresses – eg

Sep 28 13:01:03 www vpopmail[20410]: vchkpw-pop3: vpopmail user not found www@:24.153.205.71
Sep 28 13:01:03 www vpopmail[20411]: vchkpw-pop3: vpopmail user not found web@:24.153.205.71
Sep 28 13:01:09 www vpopmail[20417]: vchkpw-pop3: vpopmail user not found web@:24.153.205.71
Sep 28 13:01:11 www vpopmail[20420]: vchkpw-pop3: vpopmail user not found web@:24.153.205.71

Annoying, but not realistically going to provide much of a security issue – most of the user names are the generic ones which aren’t actually in use on the servers.

As we already use fail2ban to perform basic service blocks against naughty script kiddie wannabee’s, why not have it block vpopmail attacks also.

Our mail error logs are located in /var/log/mail.log

As you saw above, the logs show the same common text for each failed login –

vchkpw-pop3: vpopmail user not found web@:24.153.205.71

A simple regex to identify that in the logs would look like this (as per the fail2ban wiki)

failregex = vchkpw-pop3: vpopmail user not found .*@:$

First step is to create a filter for fail2ban.

Create /etc/fail2ban/filter.d/vpopmail.conf as below:

# Fail2Ban configuration file for vpopmail
#
# Author: Lawrence Sheed
#
# $Revision: 1.0 $
#

[Definition]

# Option: failregex
# Notes.: regex to match the password failures messages in the logfile.
# Values: TEXT
#
failregex = vchkpw-pop3: vpopmail user not found .*@:$

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex = 

Second step is to add our filter to the fail2ban setup

Add this to the bottom of /etc/fail2ban/jail.conf

[vpopmail]
enabled = true
port    = pop3
filter  = vpopmail  
logpath = /var/log/mail.log
maxretry = 3

logpath should be amended to whatever your mail logs for vpopmail appear.
maxretry should be set to a value that you agree with.

Restart fail2ban with a: /etc/init.d/fail2ban restart
and check that it has added the filter.

tail /var/log/fail2ban.log

You should see a line like this:

2009-10-01 12:36:09,590 fail2ban.jail   : INFO   Jail 'vpopmail' started

If so, you’re all set!

Some additional tips, as I have found some issues subsequently in Fail2ban on some systems:

If you find that fail2ban gives error 200 or 400 on occasion, this is due to a timing issue bug in fail2ban.
There are 2 possible solutions:

Solution 1 – Edit fail2ban

Open /usr/bin/fail2ban-client

Look for

def __processCmd(self, cmd, showRet = True):
beautifier = Beautifier() for c in cmd:

After for c in cmd: add a delay
time.sleep(0.5)

This should look similar to this now –

def __processCmd(self, cmd, showRet = True):
beautifier = Beautifier() for c in cmd:
time.sleep(0.5)

Save, and restart fail2ban. If you still see 200 or 400 issues, increase the delay higher e.g. time.sleep(0.8)

Solution 2 – Use a different block method

Instead of iptables, we can configure fail2ban to use route

Add a config file for this:

pico /etc/fail2ban/action.d/route.conf

Add this into the file and save it.

# Fail2Ban configuration file
[Definition]
actionban = ip route add unreachable 
actionunban = ip route del unreachable 

Open /etc/fail2ban/jail.conf

Look for ban action = … in the [DEFAULT] section, and comment it out with a # at the start of the line
then add
eg

#banaction = iptables
banaction = route

Save the file.
Restart fail2ban

It will now use route to block bad ip’s.

While I’m more of a 回力 (Hui Li) sneaker fan – I own about 5 pairs now!, I will give Kudos to Nike, for this well made video for whatever sneakers they’re pimping out these days.

Shot in and around my neighbourhood (from all the locations that I’m spotting, and going, wait a second!), it features a man running around town with nothing but a pair of…

You’ll have to watch the video for more:

This rather well done song and video about swine flu (aka H1N1) has been doing the rounds on the Chinese sites that I frequent.

As Shanghai Tattoo says “Hopefully this means the end now that it’s made it to pop culture”.

猪流感之歌 lyrics below.
Now sing along – Zhuuuuuuuuu!

Sung by:欧子
Words by: 每子爱

猪，你的流感惊天动地　　
　　感冒时的你吓得我不敢呼吸
　　
　　猪，都怪你的外国兄弟　　
　　他们的喷嚏害的我们躲避
　　
　　猪，你的体质原来那么差　　
　　害的连累我们都不知该吃啥
　　
　　猪，都怪你吃了就睡啊　　
　　这才鼻涕哗啦被老天惩罚
　　
　　啊……
　　
　　天蓬元帅这次又闯祸啦　　
　　他的喷嚏一打害人家
　　
　　每天他让人担心又害怕　　
　　他的肉啊　　
　　我不敢吃啦
　　
　　猪，我知道你也不容易　　
　　其实你的肉贵　　
　　咱早已吃不起
　　
　　猪，我帮你穿上棉大衣　　
　　你若感冒生病　　
　　咱负担不起
　　
　　天蓬元帅这次又闯祸啦　　
　　他的喷嚏一打害人家
　　
　　每天他让人担心又害怕　　
　　他的肉啊　　

Crappy google auto-translation for those who don’t read Chinese as well as I *obviously do*.

Cough. Cough, ahem, sorry a bit of H1N1 there…

Lyrics:
Pigs, your earth-shattering influenza
When you are scared of the cold I can not breathe

Pigs, blame your foreign brother
Sneeze their victims, we avoid

Pig, you had such a poor physical
We are all victims do not know the result in the吃啥

Pigs, blame you eat and sleep ah
Raining Cats and the nose is punished by God

Ah … …

Marshal canopy you get into trouble this time
Sneeze a dozen of his victims home

He worried about every day they are worried
Ah his meat
I dare not吃啦

Pigs, I know you is not easy
In fact, your meat
Our already can not afford to eat

Pig, I help you put on cotton coat
Cold if you fall ill
God can not afford

Marshal canopy you get into trouble this time
Sneeze a dozen of his victims home

He worried about every day they are worried
Ah his meat