Unfortunately, its back to the techie stuff for a few posts!

Here are my crib notes on installing NGinx on one of our client servers.

Add spawn-fcgi (cos its split from litettpd now) http://redmine.lighttpd.net/projects/spawn-fcgi/wiki/SVN


cd /downloads
svn co svn://svn.lighttpd.net/spawn-fcgi/trunk spawn-fcgi


No svn..grrr


apt-get install subversion subversion-tools
svn co svn://svn.lighttpd.net/spawn-fcgi/trunk spawn-fcgi
cd spawn-fcgi
./autogen.sh
./configure
make
make install


spawn-fcgi should be happily installed in /usr/local/bin/spawn-fcgi now

make sure we have php5-cgi
apt-get install php5-cgi

Check spawn-fcgi runs –
In my case not, had to rebuild eAccelerator @#$@#$!, did that, and all ok


/bin/spawn-fcgi -f /usr/bin/php5-cgi -a 127.0.0.1 -p 53217 -P /var/run/fastcgi-php.pid


Note port 53217 can be any unused port from some high unused number though to 65535
We’re going to be installing NGinx from debian packages, but probably better from source long term…

apt-get install nginx

pico /etc/nginx/nginx.conf


change some default settings


user www-data;
worker_processes 1;

Now we add our virtual hosts in /etc/nginx/sites-available


pico /etc/nginx/sites-available

Lastly, we add our fast-cgi settings, using our port from above 53217


fastcgi stuff here


If anyone wants to mess around with the settings for the Huawei eHome router EchoLife HG522-c (typically the ones supplied with the “3M or 4M” connection), then here are the user / pass settings.

Site: http://192.168.1.1/
User: telecomadmin
Pass：nE7jA%5m

Useful if you want to rejig the QoS settings.

If that login doesn’t work, try this – which is usually seen on the HG226 models –
Site: http://192.168.1.1
User: fiberhomehg2x0
Pass: hg2x0

The other standard modem HG520S is easier – admin / admin

Might be useful for some folks. I’m mostly posting here for myself, as I’ll probably forget and need to google it later.

[Update 23/Oct/09: Hotmail has fixed this issue now]

Our logs were showing lots of repeated send failures from Hotmail.
A closer investigation of the issue has revealed that Hotmail has suddenly decided that the mail RFC’s are too good for them to follow.

RFC’s are the standards which define how things work. When people don’t follow the standards, this makes things break.

In this case, it meant that all mail from Hotmail was being rejected, this is a Hotmail is broken issue!
Getting Hotmail to change their broken setup is likely to be non-productive – there are already a few pages of complaints about it on their site, complete with the boilerplate totally useless replies from drones who don’t understand the issue, despite it being helpfully spelled out for them.

See here –
http://windowslivehelp.com/community/p/127432/474962.aspx
http://windowslivehelp.com/community/t/123986.aspx

Unfortunately, while bouncing invalid email content is correct from a technical perspective, our clients need to be able to receive mail from Hotmail.
As an interim solution, I’ve patched qmail to allow for bare linefeeds.

This was fairly easy – a small patch to qmail-smtpd.c, a recompile, then restart qmail-smtpd.

To patch, look for switch(state) in qmail-smtpd.c, and remove the straynewline(); calls, so that barelinefeeds are accepted.
Code to change below:

case 0:
if (ch == '\n') { state = 1; break; }
if (ch == '\r') { state = 4; continue; }
break;
case 1: /* \r\n */
if (ch == '.') { state = 2; continue; }
if (ch == '\r') { state = 4; continue; }
if (ch != '\n') state = 0;
break;
case 2: /* \r\n + . */
if (ch == '\n') return;
if (ch == '\r') { state = 3; continue; }
state = 0;
break;


As the Wiki for fail2ban is a little less than explanatory than it could be (and they reversed my edits which made the instructions clearer), here are my own notes on setting up fail2ban to block pop3 attacks.

Have been seeing sample dictionary attacks on some servers for a while now from random ip addresses – eg

Sep 28 13:01:03 www vpopmail[20410]: vchkpw-pop3: vpopmail user not found www@:24.153.205.71
Sep 28 13:01:03 www vpopmail[20411]: vchkpw-pop3: vpopmail user not found web@:24.153.205.71
Sep 28 13:01:09 www vpopmail[20417]: vchkpw-pop3: vpopmail user not found web@:24.153.205.71
Sep 28 13:01:11 www vpopmail[20420]: vchkpw-pop3: vpopmail user not found web@:24.153.205.71

Annoying, but not realistically going to provide much of a security issue – most of the user names are the generic ones which aren’t actually in use on the servers.

As we already use fail2ban to perform basic service blocks against naughty script kiddie wannabee’s, why not have it block vpopmail attacks also.

Our mail error logs are located in /var/log/mail.log

As you saw above, the logs show the same common text for each failed login –

vchkpw-pop3: vpopmail user not found web@:24.153.205.71

A simple regex to identify that in the logs would look like this (as per the fail2ban wiki)

failregex = vchkpw-pop3: vpopmail user not found .*@:$

First step is to create a filter for fail2ban.

Create /etc/fail2ban/filter.d/vpopmail.conf as below:

# Fail2Ban configuration file for vpopmail
#
# Author: Lawrence Sheed
#
# $Revision: 1.0 $
#

[Definition]

# Option: failregex
# Notes.: regex to match the password failures messages in the logfile.
# Values: TEXT
#
failregex = vchkpw-pop3: vpopmail user not found .*@:$

# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex = 

Second step is to add our filter to the fail2ban setup

Add this to the bottom of /etc/fail2ban/jail.conf

[vpopmail]
enabled = true
port    = pop3
filter  = vpopmail  
logpath = /var/log/mail.log
maxretry = 3

logpath should be amended to whatever your mail logs for vpopmail appear.
maxretry should be set to a value that you agree with.

Restart fail2ban with a: /etc/init.d/fail2ban restart
and check that it has added the filter.

tail /var/log/fail2ban.log

You should see a line like this:

2009-10-01 12:36:09,590 fail2ban.jail   : INFO   Jail 'vpopmail' started

If so, you’re all set!

Some additional tips, as I have found some issues subsequently in Fail2ban on some systems:

If you find that fail2ban gives error 200 or 400 on occasion, this is due to a timing issue bug in fail2ban.
There are 2 possible solutions:

Solution 1 – Edit fail2ban

Open /usr/bin/fail2ban-client

Look for

def __processCmd(self, cmd, showRet = True):
beautifier = Beautifier() for c in cmd:

After for c in cmd: add a delay
time.sleep(0.5)

This should look similar to this now –

def __processCmd(self, cmd, showRet = True):
beautifier = Beautifier() for c in cmd:
time.sleep(0.5)

Save, and restart fail2ban. If you still see 200 or 400 issues, increase the delay higher e.g. time.sleep(0.8)

Solution 2 – Use a different block method

Instead of iptables, we can configure fail2ban to use route

Add a config file for this:

pico /etc/fail2ban/action.d/route.conf

Add this into the file and save it.

# Fail2Ban configuration file
[Definition]
actionban = ip route add unreachable 
actionunban = ip route del unreachable 

Open /etc/fail2ban/jail.conf

Look for ban action = … in the [DEFAULT] section, and comment it out with a # at the start of the line
then add
eg

#banaction = iptables
banaction = route

Save the file.
Restart fail2ban

It will now use route to block bad ip’s.

We’re seeing a huge recurrence of spam thats been getting through our spam filters., all coming from @live.com addresses.
I hadn’t seen any personally until one of our clients brought up the fact that she was receiving 20-30 sex related spam a day, all coming from Random name @live.com addresses.

A check of the logs showed that we’ve received at least 100,000 of these spam mails over the last month that have gotten through to our users.
This is something I’d obviously like to remedy.  Not receiving, processing, or storing that much spam free’s up the servers for other things.

As the number of valid addresses using @live.com accounts appears to be minimal (I could only see a handful of legitimate users sending from that domain), I have taken the decision to block any email from the @live.com domain until Microsoft can resolve their spam issues.

If you do have clients using @live.com addresses, you will be able to send email to them, but not receive from them.
We apologize for the inconvenience, but unfortunately there is no other solution that easily mitigates the issue, other than completely blocking them.

For a more technical explanation of whats happening, read below:
Read more »

One of our clients sent us an email this morning letting us know that they couldn’t send an email to a client.
They forwarded the bounce message to us (below)

12.149.35.75 does not like recipient.
Remote host said: 554 Service unavailable; Client host [usa.computersolutions.cn] blocked using Barracuda Reputation; http://bbl.barracudacentral.com/q.cgi?ip=72.51.39.20
Giving up on 12.149.35.75.

Simple enough – we’re getting blocked by Barracuda Reputation, so off I go to the link to see why.

Sorry, your email was blocked

We are sorry you have reached this page because an email was blocked based on its originating IP address having a “poor” reputation. The “poor” reputation may have been caused by one of the following reasons:

* Your email server contains a virus and has been sending out spam.
* Your email server may be misconfigured.
* Your PC may be infected with a virus or botnet software program.
* Someone in your organization may have a PC infected with a virus or botnet program.
* You may be utilizing a dynamic IP address which was previously utilized by a known spammer.
* Your marketing department may be sending out bulk emails that do not comply with the CAN-SPAM Act.
* You may have an insecure wireless network which is allowing unknown users to use your network to send spam.
* In some rare cases, your recipient’s Barracuda Spam Firewall may be misconfigured.

A quick check of our ip space over at a more legitimate place shows we’re fine – http://www.senderbase.org/senderbase_queries/detailip?search_string=72.51.39.20

I double check with a rbl lookup over here – http://www.mxtoolbox.com/blacklists.aspx, nope, we’re clean as a whistle.

However, on the same page, they have an big button helpfully letting us know that:

Many Barracuda Spam & Virus Firewalls are configured, as a policy, to automatically deliver email that comes from sources that are properly registered at EmailReg.org.

Ok, so follow the link through to EmailReg.org, and sign up.
Looks good until we get to the – a $20 fee will be charged per domain per year.

Hmm, so email will possibly be blocked by Barracuda unless I pay them $20 a year.
Sounds like Blackmail to me.

I also note that although EmailReg.org appears to be a separate entity, it is in fact owned by Barracuda. So a neutral third party blocking service just so happens to be owned by the people doing the blocking. If thats not a conflict of interest, I don’t know what is!
This is actually illegal in some countries, although apparently, not the USA.
It also doesn’t stop actual spammers coughing up money, and getting greenlisted.

Seems the rest of the net agrees with us on this one.

Quote from Mike E. that pretty much sums it up: I feel compelled to add this. If I’m paying Barracuda for a appliance to filter out spam and they in turn are being paid by spammers to allow their messages through my spam firewall, how is that different than an antivirus company taking money from somone that write viruses to have their product not detect a virus? None. It’s slimy.

So, in future when clients are unable to send mail to people using Barracuda firewall devices, I’ll be able to point them to this post, and let them know the situation.

We don’t like spam either, and work hard to avoid clients misusing our services.
However, we don’t blackmail senders into paying us money to accept their mail.

For a rundown on legitimate practices, read this:

http://en.wikipedia.org/wiki/Anti-spam_techniques_(e-mail)

Further References/Complaints:
http://www.nabble.com/zen.spamhaus.org-td22805806.html
http://www.debian-administration.org/users/simonw/weblog/295
http://zacharyozer.blogspot.com/2008/10/worst-engineers-ever.html
http://andrew.triumf.ca/barracuda-problems.html
http://community.spiceworks.com/topic/32502
http://steve.heyvan.com/2008/11/06/technology-reviews/barracudacentral-another-blacklist-black-hole/
http://ithelp.ithome.com.tw/question/10013491?tab=opinion (Trad Chinese)
http://www.linux.com/feature/155880

http://www.emailreg.org/

Had a client over today with some Mac issues.
Was getting disk full messages, despite having 130Gig free.

Did the usual stuff – disk repair, disk verify (caught some small things).
That fixed the disk full messages.

Then the client told me – oh, by the way, Safari doesn’t open.

Tailing the system log in console still revealed issues.

14/04/09 09:07:43 com.apple.launchctl.System[2] could not fetch history: Cannot allocate memory
14/04/09 09:07:43 com.apple.launchctl.System[2] BootCacheControl: could not stop cache/fetch history: Cannot allocate memory

A quick google of that error showed it was caused by…. Wacom drivers.

While Wacom have updated drivers for other tablets, the client uses a Bamboo, which hasn’t had driver updates since 2007.

Checking the logs while opening Safari revealed that it was trying to open a non-existent file called com.pentablet.defaults.xml

On the off chance that this would work, I created a blank file in terminal.

Terminal
sudo su
[enter in your password]
cd /Library/Preferences
ls -al com.pentablet.defaults.xml

If (and ONLY if) no file is found, do this:

echo > com.pentablet.defaults.xml
exit
exit

Safari will open again.

Hopefully Wacom will release newer less buggy drivers sometime soon.

As I’ve spent the day doing some pre-emptive maintenance type stuff on our servers, I noticed that one of our servers appeared to have way slower ping times than normal from our other servers.

Taking a closer look I also saw that the logs said it having timeout problems talking to our database server.

A quick look at the server logs showed that the server in question was using 100% of its bandwidth for the last hour or so according to Monit.

A quick check with ntop (excellent web based network analysis statistics) and iftop (console based network traffic analysis) confirmed that 99.5% of the bandwidth was going to HTTP requests.
Strangely enough, the server wasn’t even stressed at all (I guess I’ve overspecced that one!)

Read more »

One of our clients was sending out spam unknowingly yesterday. I spent most of my afternoon cleaning it up, tracking down how the attackers were doing it.

In this clients case, they have their own server (which we maintain), and they mostly write their own code.
Most of the common garden variety vulnerability scans don’t work on their server, because they write their own code, although in this case it didn’t save them from being exploited.

In order to find out what was causing the spamming, I had to find out how the attackers got in.
Usually this means a check of the apache logs to check for anything untoward.

In this case, although the logs had plenty of vulnerability scans (which were to files that don’t exist on their server), I couldn’t see anything in the logs that immediately stuck out as being the cause.
Read more »

Foreword – Note that none of our servers are vulnerable to remote inclusion attacks.

For the most part, most of the exploits I covered in yesterdays post are common garden php vulnerability scans.
Some of them are more interesting though, although more for being encrypted, than anything else.

If I take an example from our log files:
Read more »