This was posted by myself over at BimmerForum’s, but as China’s Firewall is being particularly unpleasant these days about what sites I can and cannot visit, I thought I’d repost it here, so its more accessible for me.
Note – I no longer own a 7 series, so any requests for modded rom’s should probably be accompanied with nice gifts to persuade me to do the work involved 🙂
As I saw people asking about Tuned ROM’s in another post I thought I’d do some research.
Its quite niche – hence the ridiculous pricing people charge for roms – time is money though!
Here’s my research so far. I’m basing it on the E32 735iL 1989/1990, as thats what I own (and can test for).
Googling for E32 735iL gives:
ELECTRONIC CONTROL UNIT, DME MOTRONIC
BOSCH 0261200179
BMW 1730697
*26SA0840*
Vehicle Applicability:
1986-1993
BMW 3′ E30 320i Touring 2.0 95kW/129PS (M20/ 206EE/KA)
BMW 5′ E34 535i 3.4 155kW/211PS (M30/346KB)
BMW 6′ E24 635CSi 3.4 155kW/211PS (M30 B35)
BMW 7′ E32 735i 3.4 155kW/211PS (M30/ 346KB/EC)
BMW 7′ E32 735iL 3.4 155kW/211PS (M30/ 346KB/EC)
ELECTRONIC CONTROL UNIT, DME MOTRONIC
BOSCH 0261200179
BMW 1722611
*26RT2794*
Vehicle Applicability:
1986-1993
BMW 3′ E30 320i Touring 2.0 95kW/129PS (M20/ 206EE/KA)
BMW 5′ E34 535i 3.4 155kW/211PS (M30/346KB)
BMW 6′ E24 635CSi 3.4 155kW/211PS (M30 B35)
BMW 7′ E32 735i 3.4 155kW/211PS (M30/ 346KB/EC)
BMW 7′ E32 735iL 3.4 155kW/211PS (M30/ 346KB/EC)
ETK however says
No.Description Supplement Qty Part Number Price
01AT-Control unit Motronic HARDWARE 1.3 1 12141738168 ENDED
01AT-Control unit Motronic HARDWARE 1.3 1 12141748258 $813.86
Whatever the actual revision used, its pretty clear we have a v1.3 Motronic.
Wiki (
) says:
The Motronic 1.1 System was used by BMW from 1987. This was then superseded in 1988 by the Motronic 1.3 system[6] that was also used by PSA on some XU9J-series engines (which previously used Motronic 4.1).[5]
The Motronic 1.1 and 1.3 systems are largely similar, the main improvement being the increased diagnostic capabilities of Motronic 1.3. The 1.3 ECU can store many more detailed fault codes than 1.1, and has a permanent 12-volt feed from the vehicle’s battery which allows it to log intermittent faults in memory across several trips. Motronic 1.1 can only advise of a few currently-occurring faults.[6] The systems include a knock sensor for ignition timing adjustment and the option for a lambda sensor, enabling their use with catalytic converter-equipped vehicles.[5] The ECUs have 2 injection outputs, and the injectors are arranged in 2 "banks" which fire once every two engine revolutions. In an example 4-cylinder engine, one output controls the injectors for cylinders 1 and 3, and the other controls 2 and 4. The system uses a "cylinder ID" sensor mounted to the cam-shaft to detect which cylinders are approaching the top of their stroke, therefore which injector bank should be fired. During start-up (below 600 RPM), or if there is no signal from the cylinder ID sensor, all injectors are fired simultaneously once per engine revolution.[6] |
DME / ECU 1.3 Info.
Tuning is set in 2D map.
Motronic 1.3 from 1989: The fault code memory is extended to contain all fault codes that are detected by the EMS.
In vehicles sold in the USA, when the ECU detects that a some faults are present it earths pin 15 and the Check Engine warning lamp on the dash will light. The lamp will stay lit until the fault is no longer present. A warning lamp is not fitted to vehicles sold in other markets. The faults that will turn on the lamp are mainly those concerned with emissions. Other faults are logged by the ECU but the lamp will remain out.
ECU (DME) sensors run off 5v
Download extended info here – http://www.opel-scanner.com/files/DME_1.1_1.3.pdf
DME bootup: (roughly paraphrased)
Interestingly enough I got to learn this properly via the tips and tricks for checking why the car doesn’t start..
Ignition switched on:
ECU earths 85 , 36 which causes relay to close and 30 is connected to 87 starting the fuel pump. (30+87 can be bridged manually under the green relay to test pump). Its opened for +- 1 second.
Next up it reads values from the CAS (Pulse Sensor). If it reads a speed signal (aka is it turning), so fuel pump stays on.
CAS is used to time the pulses (read from each turn of the flywheel), this is checked against the map file for correct speeds (need to use a multiplier for the map values vs the flywheel pins).
The Motronic ECU contains a fuel map with an injector opening time for basic conditions of speed and load. Information is then gathered from engine sensors such as the AFS, CAS, CTS, and TS. As a result of this information, the ECU will look-up the correct injector pulse duration right across the engine rpm, load and temperature range.
The injectors are arranged in two banks with injectors 1 and 3 (4 cylinder) or 1, 3 and 5 (6 cylinder comprising one bank, and injectors 2 and 4 (4 cylinder) or 2, 4 and 6 (6 cylinder) making up the other bank. Each bank is connected to the ECU via an independent ECU pin.
The Motronic 1.1 & 1.3 multi-point injection system pulses the injectors semi-sequentially and once every two engine revolutions. During engine start-up below 600 rpm the ECU pulses all injectors simultaneously. Once 600 rpm has been attained and if the ECU has received a signal from the CID sensor, each injector bank will be pulsed alternatively according to which pair of cylinders are approaching TDC. If a signal is not received from the CID sensor the injectors will remain on simultaneous operation. However, if the CID sensor subsequently sends a signal to the ECU after the engine has commenced running, the ECU will pulse the injectors semi-sequentially after the next deceleration phase – even if the CID sensor then ceases to send a signal.
—
Motronic troubleshooting (aka code readout)
The fault codes for these systems will appear on the check engine light when you turn the ignition key to the On position (but dont start it), then: fully depress the accelerator pedal, then fully release it. Repeat the depress,release cycle 5 times fairly quickly, but not too fast. It should be completed in a few seconds. What you are doing here is clicking the wide open throttle switch then the idle switch each 5 times. This signals the Motronic unit to send the fault codes by flashing the Check Engine light. You should see the light flash once then it will start flashing the 4 digit codes. If there are no faults, you should see the 1444 code (and 2444 if you have a 12 cylinder). The codes appear as a series of flashes for each digit. The flashes indicating one digit are about 1 second apart, the next digit appears after a couple second interval.
Get the codes from: http://www.unofficialbmw.com/repair_faqs/motronic.html
Other notes:
CPU most likely 8051
EEPROM 26C256, so 32KB / (32768 byte) chip. 28 pin.
Can also use a 64KB chip, just ground the relevant pins for which 32kb to use or burn one rom at appropriate location. Useful for running 2 tunes – eg original and second.
Tools available for this eg http://www.moates.net/font-size-12ti….html?cPath=31
Should run from a AT29C256 or similar.
ECU interesting pinouts:
Pin 55 TX
Pin 13 RX
Pin 2/19/24/26/14 GND
File layout (unchecked)
AFR idle: 0x662E (pos 26158)
AFR partial: 0x6662
AFR full: 0x6640
ign idle: 679A
ign partial: 0x67FE
ign full: 0x67CC
Sample bin file here: ftp://anonymous@bmw-wiki.org/ROMs/Mo…y-Motronic.bin
(I’m having issues downloading that from China, so haven’t checked it yet against the alleged layout).
Table calculations:
fuel maps correspond to (value/128 +1)*lambda_1
Bosch-Motronic f(n) = (n-30) x 0.75
some Bosch f(n) = (n-509) x 0.75
Bosch KE-Motronic und Mono-Motronic
f(n) = [(255-n)/6] +1
Motronic info
Very useful info for looking this up from http://motronic.ws/map201.htm
Maps 20 bytes each
Other thoughts – now we know what pinouts / values are expected in the system, guess just need to see what equipment provides appropriate inputs. eg, replace expensive spares with cheaper/better modern versions.
Most of the differences are usually in the mounting afaik, not the tech.
Motronic pinouts and board details here – http://www.e34.de/tips_tricks/motronic/m1_3.pdf
Lots of people working on this kind of thing with software like tunerpro / xdf (map) files.
*Good* forum for info (although not relevant to the Bosch ECU’s, but concepts / strategies same)
http://www.thirdgen.org/techboard/di…uide-book.html
The motronic 1.3 software in most of the E32’s should be fairly easy to modify – its all been documented well online before, so just a matter of getting a few rom files for specific models, and comparing tables.
Shogun has at least one rom (as he’s burning and selling them according to another post).
I can extract the current one from my DME and make a bin file for comparison.
We also have the rom from ftp://anonymous@bmw-wiki.org/ROMs/Mo…y-Motronic.bin
Should be enough to see where the changes are for the E32 / 8051CPU Motronic 1.3 anyway, so that I can tweak.
Another lucky thing is that I also used to write embedded software on 8051 cpu years ago (although I’ve forgotten it all by now)
Any interest in this from others (I’ll need more rom’s though)?
As my car currently has only 1 key, I thought I’d do a little bit of research into getting another one.
BMW quoted 2000rmb, which although understandable, is a little bit expensive. That doesn’t include the remote functionality in the current key either.
A check of taobao shows that you can buy blanks for slightly less – in the 30rmb range.
Mine is a 3 button remote, rather than a 2 button remote.
Images of both below:
Yes, I realise that these are just keys, and don’t contain the remote stuff either, but as a spare, its all I really need.
There is a fly in this ointment though. All European cars post 94 are mandated to include anti-theft immobilizers, which in BMW’s case involves checking for a transponder, so a key won’t start the car.
BMW calls this system EWS – Electronic Watchdog System.
A datasheet aimed at BMW workshop staff on EWS is here BMW EWS Overview and Worksheet
The infrastructure looks something like this:
I could buy a small board to bypass the EWS (about 130rmb), but I like the idea of having the anti-theft still working.
So, how does the EWS2 do the immobilization in my car?
A little bit of research shows that my current BMW (a 96 E36 import from German) uses BMW’s EWS2 for antitheft. This appears to be based on a motorola chipset ( XC424114CFN ) for the immobilizer side, and its relatively easy to read out or reflash the existing data from the chip if necessary should I need to do so.
eg if I lost all my keys, and needed to start from scratch. There are 3rd party tools galore that go into that eg CarProg, so not much point looking at that for me.
Lets look at the transponder side.
The transponder in my current key is a Philips PCF7930
This comes in a series – PCF7930, PCF7931, and PCF7935, and PCF7936
The main differences between the transponders are as follows:
PCF7930 – can read / write data.
PCF7931 – write once (one time programmable), read data.
PCF7935 – it has 2 area’s of memory general, and shadow memory, and both are read / write.
PCF7936 – it adds a crypto mode to the 7935 functionality.
The PCF7936 is also referred to as a Hitag2.
This is also the transponder current generation of BMW’s are using eg the E90/E92/E94’s
They’re quite simple devices really, a datasheet for them is here – PCF7930 / 31 / 35 Datasheet
They essentially have a few bytes of memory to work with.
32 bytes for control, and 80 bytes for user data.
Physically they’re about half the size of a fingernail and a few mm thick.
They look something like this –
A quick search on taobao shows that a PCF793x sells for between 17-20rmb or so.
So, so far I can get the key for about 30 odd, and the transponder for about 20rmb or so.
I still need to be able to program the transponder though, so whats available for that?
Another quick google shows that there is a common windows tool called anahtar which works with quite a few programmers.
As you can see, it supports quite a few pieces of hardware. Anahtar does need some hardware to talk to the transponder though, so I also need a transponder programmer.
A search for a normal RFID programmer is pretty polluted with car remote programmer spam sites.
That said, the usual result for BMW’s is the AK90 programmer. This is a bit on the expensive side – its around 1500rmb, and I really only need to do this once, so lets look for other options.
Ideally we’re looking for a low frequency rfid programmer.
If possible, I’d like to have something that does other chips too, so something that can cope with the below may be handy at some point, if only so I can steal peoples cars play with it.
RWProg looks interesting, as it has a lot of support for other rfid chips – http://www.bicotech.com/?page=prod_rwprog&lg=en.
Unfortunately a search for that on Taobao shows no results. RFID reader’s on the other hand are dirt cheap, with the cheapest usb ones going for 35rmb or so.
The specs for the PCF793x series don’t actually say what frequencies it runs off unfortunately, so its a little hard to find an appropriate device quickly.
I do note that there appear to be a disproportionate amount of card writers advertised on Taobao which claim to do Mifare, which not co-incidentally is similar enough to what is being used on the metro here in China in most cities for travel cards. I guess that means there probably are lots of fake ones around…
If I check whats usually used for programming the PCF793x series, Philips (NXP) pushes their PCF7991, while Philips doesn’t write what frequency that runs on either, this chinese site says 125khz http://www.docin.com/p-74627587.html
So, its a 125khz programmer.
Unfortunately those are 500bucks on tabao, so I keep looking.
…and Bingo, I can find a BMW key programmer which will do it for 350rmb.
Sounds good.
350 for the transponder reader/writer, 20 odd for the transponder, and 30 odd for a key.
400RMB total, and new copies of the keys will cost about 50rmb each vs the 2000rmb bmw wants.
Its a win!
Next up, what frequency does the remote use, so I can get a replacement for that…
My remote frequency is 433.92Mhz, as I have a Europe car.
I’ll guess it would probably be 315MHz if it was a US, CN or SA built car.
As I’ve had some negative comments about this all being possible, added some photos of the tool I bought to successfully clone my keys.
Key Cloning tool MCU adaptor side view
Cloned keys (original on far left)
Software Side:
Installing hardware drivers
Device is a standard FTDI usb serial chip based product, needs 2 drivers installed, the first for the serial chip, then a second for the device driven by the usb to serial chip.
Installing software
Software running
22
Why we love China so…
Had a client ask about some .cn stuff over Skype, so proceeded to give him the whole this is what is needed now haha fun story
i.e but no, theres more, you also need this…. and this…. and this… and don’t forget the photo in front of the sign..
While we do have our own list of requirements in this blog, I did google to see if there was something less dry, and discovered a little blog post which mirrored our own experiences quite similarly. Its a good read too –
Read this http://www.candisgroup.com/blog/busdev/the-history-of-chinese-icp-and-cn-ownership-2, then follow on below, as this was intended to be posted as a comment on that post, but their blog comments appear to be borked at the moment.
I’ve fleshed it out a little, and it still seems semi relevant here.
Do remember that while anything I say can and will be used against me as evidence, I never said it. Got that? Good, now read on.
—
Glad to see I wasn’t the only one that had to go through all that, and again in October last year.
I also fondly remember the shut down their site stuff.
First we made a ICP warning page, then a blank page, and then I got creative and started mapping the IP for the domain to the miibeian.gov.cn server – you never know, maybe they’ll shut down their site!
Apparently although this was amusing, it also wasn’t enough, and we too had to vanish the DNS into nothingness NOW
(which is also hard, as explaining that things take 1-2 days to propagate doesn’t seem to sink in with the telecom people)
Mind you, the threat from the terrori^H Telecom people to shut off their site didn’t really have the same effect when its already shut off dammit!
Then there are the clients who ask why do we have 4 icp addresses?
Well, because the government decided that you could only have 1 per login when we did them in 2005. However client just has 1 site, and they point .com .com.cn .cn et al to the same place. and Telecom INSISTS that you have an ICP for all of them, even though its the same site. Luckily they seem to have fixed that now, although we still have the odd site with 3 or 4 icp’s just because you can’t seem to get them to accept they’re all the same site dammit, just let us register them to the one ICP, please!
…and then there are the mysteriously disappearing ICP numbers of doooom, where the telecom people says that ICP is invalid. Its fake I tell you, FAKE!
Although you could swear that the ICP is real as you did all that darn paperwork the last time around, plus you have an email from miibeian which has that number, and domain, so please please recheck… but they still say thats impossible, it doesn’t exist in their system.
Why, why, why, would we possibly have a bazs.cert for the domain together with the icp and the url tucked away in our files if we didn’t receive it from them?, but no, we have to do it all again because their system is buggy, and these aren’t the droi^Hnumber’s you’re looking for.
At least they seem to have dropped that requirement now for the bazs.cert file to be uploaded to all sites now. I shouldn’t say that too loud though, someone up high will probably smack their head now, and say, I knew there was some other hoop we forgot to get people to jump through now, erm, I mean new legislation thats not clearly thought out so people have to work out the hard way what we want, er no, thats not right either, forget what I said…
Or the good old China optimistic scheduling / planning with regards to implementation dates?
i.e. You have 2 weeks to get all of your 700+ client registrations completed in triplicate, filed with CNNIC, registrar, and local telecom authority or we shut their site off…. What do you mean you need some kind of notice.
Jump NOW! We say how high.
Ah, the good old days.
At least the clients are understanding when you tell them hey, the government changed the rules again, we need more stuff, come for photos, bring your id’s, and business licences, and do it asap, or you lose the domain.
It’s at time like this when I think back to what should have been the slogan for the Shanghai expo and the thing with 5 circles up in Beijing (just prior to the big bonfire and marshmallows thing at the CCTV building that I didn’t get invited to).
China. – We invented bureaucracy.
23
IPCam Hacking Part #7
Its been a while since I did any IPCam stuff, but I’ve now got most of the bits I needed together again, as well as a new laptop for dev work (curse the thieves that stole my last one!)
As we recall from previous work, the main binary for the IPCam runs off a file called “camera”.
As some people have discovered, it likes to reboot the equipment when its not happy (eg when the camera is unplugged, it has issues talking to hardware, or when someone has flashed the wrong firmware).
So, lets take a look at the executable to see what interesting bits we can find out from it.
#file camera tells us – BINFLT file format. Fileflags: RAM GZIP.
So we know its a compressed bflt elf – bflt stands for binary flat file, and it uses gzip compression. It also sits in ram.
A hex dump of camera shows this for the first few bytes:
62 46 4C 54 00 00 00 04 | bFLT . . . .
[Note – I had about 4 pages of #$%# work done on this, and WordPress decided to flake once finished due to an errant pasted 0x0 null byte above, cutting off the rest of my post, so this is going to be shorter and angrier than it was originally written.
Lesson learned, always save stuff elsewhere before posting.]
bFLT headers consist of 4 bytes identifier, then 4 bytes for the version number.
In this case, its a version 4 bFLT file.
If we take a look at the header file source for bflt at the uclinux site we see the below layout.
struct flat_hdr {
char magic[4];
unsigned long rev; /* version */
unsigned long entry; /* Offset of first executable instruction with text segment from beginning of file */
unsigned long data_start; /* Offset of data segment from beginning of file */
unsigned long data_end; /* Offset of end of data segment from beginning of file */
unsigned long bss_end; /* Offset of end of bss segment from beginning of file */
/* (It is assumed that data_end through bss_end forms the bss segment.) */
unsigned long stack_size; /* Size of stack, in bytes */
unsigned long reloc_start; /* Offset of relocation records from beginning of file */
unsigned long reloc_count; /* Number of relocation records */
unsigned long flags;
unsigned long filler[6]; /* Reserved, set to zero */
};
It doesn’t match up properly, as the sizes or code don’t make sense (yet).
If we take a closer look, the header file has this to say:
#define FLAT_FLAG_RAM 0x0001 /* load program entirely into RAM */
#define FLAT_FLAG_GOTPIC 0x0002 /* program is PIC with GOT */
#define FLAT_FLAG_GZIP 0x0004 /* all but the header is compressed */
Ahah!
So, all but the header is compressed for a version 4 file.
Lets check this out, and see if its correct.
Excluding the initial file identifier (bFLT), our header consists of 10 longs. Thats 40 bytes long.
Lets jump to offset 40 in the file, and see what we have there.
1F 8B 08
Those of you familiar with gzipped files will recognize that – its the gzip header identifier. So, so far, so good.
Compressed files aren’t very useful to us, as they don’t show much text content.
So, we could unzip the file to take a look at whats inside.
There are a number of ways we can unzip this (zcat, gzip -d etc), but I’m going to be lazy, and use someone elses premade code.
See below for some perl to safely uncompress our binary , taken from here – http://www.openwiz.org/wiki/BWFWTools_Release
#!/usr/bin/perl
=pod
=head1 NAME
gunzip_bflt - convert gzip-compressed bFLT executable files into uncompressed bFLT
=head1 SYNOPSIS
gunzip_bflt zipped_blflt_files...
=head1 DESCRIPTION
Convert gzipped bFLT files into an uncompressed bFLT files.
The unzipped bFLT files have B<.unz> added to their file names.
If the file is already ungzipped bFLT, it isn't converted,
but a warning is printed.
=head1 PREREQUSITES
Uses packages C
=cut
use strict;
use warnings;
# gunzip_bflt zipped_blflt_files...
use IO::Uncompress::Gunzip qw/gunzip $GunzipError/;
use POSIX;
# Read and return the BFLT header
# prints a warning and returns undef on error.
# $bfltZfh is the BFLT file handle,
# $bfltZ is the BFLT file name (for error messages)
sub get_bflt_hdr($$) {
my ($bfltZfh, $bfltZ) = @_;
my $buf;
my $res = sysread $bfltZfh, $buf, 64;
if(!defined($res)) {
warn "$bfltZ: $!\n";
return undef;
}
if($res < 64) {
warn "$bfltZ: Too short!\n";
return undef;
}
# Align the buffered file handle with the unbuffered
seek $bfltZfh, sysseek($bfltZfh, 0, SEEK_CUR), SEEK_SET;
return $buf;
}
# Expand a gzipped BFLT intoi an ungziped BFLT
sub expand_blftZ($) {
my ($bfltZ) = @_;
my $bflt = $bfltZ . '.unz';
if(!open BFLTZ, '<' . $bfltZ) {
warn "$bfltZ: $!\n";
return;
}
my $hdr = get_bflt_hdr(\*BFLTZ, $bfltZ);
if(!defined $hdr) {
return;
}
if(substr($hdr, 0, 4) eq 'bFLT') {
# Pack/unpack template for the BFLT header, 4 bytes ACSII,
# 15 little-endian words
my $hdrFmt = 'a4 N15';
my @unpHdr = unpack $hdrFmt, $hdr;
# Test the header flags 'gzipped' bit
if($unpHdr[9] & 4) {
# Unset the header flags 'gzipped' bit, and make a new header
$unpHdr[9] &= ~4;
$hdr = pack $hdrFmt, @unpHdr;
if(open BFLT, '>' . $bflt) {
# Write the header
syswrite BFLT, $hdr;
# Align the buffered file handle with the unbuffered
seek BFLT, sysseek(BFLT, 0, SEEK_CUR), SEEK_SET;
# Ungzip from the compressed file into the uncompressed
# file
gunzip \*BFLTZ => \*BFLT
or die "gunzip failed: $GunzipError\n";
close BFLTZ;
} else {
warn "$bflt: $!\n";
return;
}
} else {
warn "$bfltZ: Not a compressed bFLT file, not gunzipped\n";
return;
}
} else {
warn "$bfltZ: Not a bFLT file\n";
}
close BFLT;
}
# Expand the arguments...
foreach my $bfltZ (@ARGV) {
expand_blftZ($bfltZ)
}
If we run that on our ‘camera’ executable, we should have an uncompressed bFLT file as output ‘camera.unz’.
Lets run strings on ‘camera.unz’, to see what interesting text content is in there.
Some interesting things of note:
From the variables list below, looks like there is a way to turn off the LED…
led_mode
ptz_center_onstart
ptz_auto_patrol_interval
ptz_auto_patrol_type
ptz_patrol_h_rounds
ptz_patrol_v_rounds
ptz_patrol_rate
ptz_patrol_up_rate
ptz_patrol_down_rate
ptz_patrol_left_rate
ptz_patrol_right_rate
We also have a full list of the internal cgi functions now, which might prove useful…
snapshot.cgi
get_status.cgi
get_camera_params.cgi
decoder_control.cgi
camera_control.cgi
reboot.cgi
restore_factory.cgi
upgrade_firmware.cgi
upgrade_htmls.cgi
get_params.cgi
set_alias.cgi
set_datetime.cgi
set_users.cgi
set_devices.cgi
set_network.cgi
set_wifi.cgi
set_pppoe.cgi
set_upnp.cgi
set_ddns.cgi
set_ftp.cgi
set_mail.cgi
set_alarm.cgi
videostream.cgi
video.cgi
test_ftp.cgi
test_mail.cgi
set_misc.cgi
get_misc.cgi
set_p2p.cgi
get_p2p.cgi
set_forbidden.cgi
get_forbidden.cgi
set_decoder.cgi
comm_write.cgi
wifi_scan.cgi
get_wifi_scan_result.cgi
get_log.cgi
check_user.cgi
check_user2.cgi
backup_params.cgi
restore_params.cgi
erase_allparams.cgi
set_mac.cgi
do_cgi: unknown cgi
You can see that Maverick decided to fake the X-Mailer smtp header (Foxmail is a commonly used Mail Program in China).
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary=”smtp_msg_boundary”
X-Mailer: Foxmail
–smtp_msg_boundary
Content-Type: image/jpeg;
name=”%s(%s)_%c%s.jpg”
Content-Transfer-Encoding: base64
–smtp_msg_boundary–
I’m interested in why the firmware reboots on some firmwares though, so lets take a deeper look at the code.
To do so, we’ll need to decompile it.
The better equipped than me will probably use something like the nice ARM decompiler plugin for IDA-Pro called Hex-Ray. Unfortunately that costs $$$, and I’m just a hobbyist.
Luckily there is a free windows decompiler called arm2html available here.
arm2html doesn’t handle compressed bFLT files though, so you’ll need to point it at the freshly ungzipped code you got from the perl script above.
As we know that the camera executable reboots after issuing i2c errors, the first piece of decompiled code I wanted to look at was the first piece of code related to i2c:
(excerpted piece below)
02588: e1a0c00d mov ip, sp
00258c: e92dd810 stmdb sp!, {r4, fp, ip, lr, pc}
002590: e24cb004 sub fp, ip, #4 ; 0x4
002594: e24dd00c sub sp, sp, #12 ; 0xc
002598: e59f023c ldr r0, [pc, #572] ; [0027dc] "/dev/i2c0"
00259c: e3a01002 mov r1, #2 ; 0x2
0025a0: eb00c9b8 bl 034c88(c9b8)
0025a4: e1a04000 mov r4, r0
0025a8: e3540000 cmp r4, #0 ; 0x0
0025ac: aa000003 bge 0025c0(3) ; jump
0025b0: e59f0228 ldr r0, [pc, #552] ; [0027e0] "%s: can not open i2c device"
0025b4: e59f1228 ldr r1, [pc, #552] ; [0027e4] "zoom_test"
0025b8: eb00bc86 bl 0317d8(bc86)
0025bc: e91ba810 ldmdb fp, {r4, fp, sp, pc} ; return
The full piece of code essentially loops 7 times trying to open the i2c sensor to call the zoom_test code. If it fails, it calls for a reboot.
Success proceeds to setting up the camera.
We know from our boot log that my camera in this model is a Sonix288.
dvm usb cam driver 0.0.0.0 by Maverick Gao in 2006-8-12
usb.c: registered new driver dvm
dvm usb cam driver 0.1 for sonix288 by Maverick Gao in 2009-4-20
usb.c: registered new driver dvm usb cam driver for sonix288
The Sonix288 is a chipset SoC that will talk to an attached image sensor via i2c. I think that the Sonix288 is probably a standard USB 1.1/2.0 compatible UVC (USB Video Class) chipset from a bit of googling about it.
We don’t have the source for our particular camera though (its secret, much like the data sheets, grrr…).
What to do?
Linux generally uses spcaxxx (and UVC) drivers for talking to camera’s, so lets start taking a look there.
http://read.pudn.com/downloads127/sourcecode/unix_linux/539050/zc030x/zc030x_cameras.c__.htm
http://www.hackchina.com/r/54654/zc030x_i2c.c__html
Taking a look at some spcaxxx driver header files and code shows that i2c is setup by first getting the USB VID, USB PID of the hardware, then talking to the i2c device on that hardware.
So, we need to know what our USB VID and PID’s are.
Generally all devices from a given manufacturer will have a single VID as issued by the USB Forum.
If we search through the code for Sonix, it appears that Sonix’s VID is 0x0c45
{USB_DEVICE(0x0c45, 0x607c)}, /* Sonix sn9c102p Hv7131R */
(from http://linux.downloadatoz.com/linux-kernel-webcams-driver-gspca-spca5xx/ )
The Linux UVC page confirms this http://www.ideasonboard.org/uvc/.
(Listings of webcams excerpted – note the VID of 0c45)
0c45:6310 USB 2.0 Camera (Trust Chat Webcam) Sonix Technology
0c45:63e0 Sonix Integrated Webcam (Dell notebooks) Sonix Technology
0c45:63ea Laptop Integrated Webcam 2M (Dell Studio 1555 notebooks) Sonix Technology
0c45:6409 USB 2.0 Camera (Nokia Booklet 3G netbooks) Sonix Technology
0c45:6415 Laptop Integrated Webcam 1.3M (Dell Inspiron 13z notebooks) Sonix Technology
Ideally at this point, I’d have a data sheet for the Sonix288 chip, but the #$%#$ people at Sonix don’t seem to publish one for us mere mortals.
So, we’ll use the next best thing, and use one for their other chipsets.
SN9C1xx PC Camera Controllers –
http://ww2.cs.fsu.edu/~rosentha/linux/2.6.26.5/docs/video4linux/sn9c102.txt
There is a lot of good useful info in that particular file. We don’t know how much is useful yet, but generally chipsets are quite similar for a given range.
Image sensor / SN9C1xx bridge | SN9C10[12] SN9C103 SN9C105 SN9C120
——————————————————————————-
HV7131D Hynix Semiconductor | Yes No No No
HV7131R Hynix Semiconductor | No Yes Yes Yes
MI-0343 Micron Technology | Yes No No No
MI-0360 Micron Technology | No Yes Yes Yes
OV7630 OmniVision Technologies | Yes Yes Yes Yes
OV7660 OmniVision Technologies | No No Yes Yes
PAS106B PixArt Imaging | Yes No No No
PAS202B PixArt Imaging | Yes Yes No No
TAS5110C1B Taiwan Advanced Sensor | Yes No No No
TAS5110D Taiwan Advanced Sensor | Yes No No No
TAS5130D1B Taiwan Advanced Sensor | Yes No No No
Interesting… Hmm, so it seems that Sonix uses a SN9Cxxx for its product names.
Lets google for SN9C288 instead, and see if we get any results.
Bingo.
From here – www.lh-invest.com/en/showpro.asp?id=308&proid=3
* MCU: SONIX SN9C288
* Sensor: MICRON K14 1,300,000 pixels CMOS
* 5-glass lens,can reach 30 frames/sec. under 640*480
resolutions,software insertion value 1,300,000 pixels
* Focus range: 3CM to infinitude farness
* Dynamic video resolutions: 1280*960 pixels(max.)
* Support microsoft UVC driver-free function
* Support RGB24 and YUY2 two kinds of image formats
* USB 2.0 port,support plug and play
* Human face tracking function
Seems our chipset is finally getting some details.
Max resolution, video modes, face tracking capability, etc
We also know that it can be paired with a Micron K14 image sensor.
Further googling reveals it can also be paired with the MI-0360 (which is listed above).
SN9C288+MI0360
This page also gives us a possible pid:
“Device Name: ?USB Video Device
PnP Device ID: VID = 0C45 PID = 62C0
Serial Number: 6&&2BCAFCF3&&0&&0000
Revision: (Information not returned)Device Type: Standard USB device – USB2.0 High-Speed
Chip Vendor: SONiX
Chip Part-Number: SN9C288PFG
In Microsoft parlance, this looks like this USB\VID_0C45&PID_62C0
Googling that gives us a product with windows drivers (HP) and more.
http://www.downloadwindowsdrivers.info/usb/vid_0c45/pid_62c0/mi_00/
Also says that these product drivers also work.
USB\VID_0C45&PID_62C0 ;SN9C211/213/230
That means we can take a look at their inf file and see if anything useful in there. Unfortunately I did, and there isn’t much 🙁
Sonix has datasheets for some of their other products available though (SN9C2028AF).
First, a quick overview of UVC devices (snarfed from elsewhere).
USB devices are required by the USB specification to respond to the Host device (the computer) with a stream of data describing the device and the interface to the device. This “Device Descriptor” includes vendor, product, and version IDs specific to the manufacturer, the product, and the version of the product. In addition to the device information, the descriptor also includes information on how to talk to the device through a series of “Interface Descriptors”.
The device descriptor for Video is 13.
#define USB_DEVICE_CLASS_VIDEO 0x0E
In addition to the end points described in the interface descriptors, all USB devices support a control pipe to end point 0. This is used to manipulate some of the low level functions of the device such as power, and error status queries.
The USB Video specification describes two interfaces, a control interface to manage the camera, and a stream interface to send or receive video information from the camera.
The control interface is used to manipulate the camera parameters such as brightness and contrast as well as to negotiate a valid set of the video format, frame size, frame rate, and compression rates parameters that describe a video stream. In addition, the control interface can ask the camera for still frame.
In the datasheet for the 2028, it basically states that they use end point 0 for STD Commands, with a maximum packet size of 64 bytes.
Further googling reveals that there is no special driver for it, its a plain UVC 1.0 device.
usb 1-2: New USB device strings: Mfr=2, Product=1, SerialNumber=0
usb 1-2: Product: USB 2.0 Camera
usb 1-2: Manufacturer: Sonix Technology Co., Ltd.
Linux video capture interface: v2.00
uvcvideo: Found UVC 1.00 device USB 2.0 Camera
Further heavy baidu’ing in Chinese sites finds that the SN9C213 and SN9C288 are the same pretty much.
其内部编号是SN9C213,功能完全和SN9C288一样。From http://ep.cbifamily.com/2007/04/44/87819.html
David McCullough very nicely also compiled in usb debug support on his kernel and ran some tests too:
> > > T: Bus=01 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#= 2 Spd=12 MxCh= 0
> > > D: Ver= 2.00 Cls=ef(unk. ) Sub=02 Prot=01 MxPS=64 #Cfgs= 1
> > > P: Vendor=0c45 ProdID=62f1 Rev= 1.00
> > > S: Manufacturer=Sonix Technology Co., Ltd.
> > > S: Product=USB 2.0 Camera
> > > C:* #Ifs= 2 Cfg#= 1 Atr=80 MxPwr=500mA
> > > I: If#= 0 Alt= 0 #EPs= 1 Cls=0e(unk. ) Sub=01 Prot=00 Driver=(none)
> > > E: Ad=83(I) Atr=03(Int.) MxPS= 16 Ivl=6ms
> > > I: If#= 1 Alt= 0 #EPs= 0 Cls=0e(unk. ) Sub=02 Prot=00 Driver=(none)
> > > I: If#= 1 Alt= 1 #EPs= 1 Cls=0e(unk. ) Sub=02 Prot=00 Driver=(none)
> > > E: Ad=81(I) Atr=05(Isoc) MxPS= 128 Ivl=1ms
> > > I: If#= 1 Alt= 2 #EPs= 1 Cls=0e(unk. ) Sub=02 Prot=00 Driver=(none)
> > > E: Ad=81(I) Atr=05(Isoc) MxPS= 256 Ivl=1ms
> > > I: If#= 1 Alt= 3 #EPs= 1 Cls=0e(unk. ) Sub=02 Prot=00 Driver=(none)
> > > E: Ad=81(I) Atr=05(Isoc) MxPS= 512 Ivl=1ms
> > > I: If#= 1 Alt= 4 #EPs= 1 Cls=0e(unk. ) Sub=02 Prot=00 Driver=(none)
> > > E: Ad=81(I) Atr=05(Isoc) MxPS= 600 Ivl=1ms
> > > I: If#= 1 Alt= 5 #EPs= 1 Cls=0e(unk. ) Sub=02 Prot=00 Driver=(none)
> > > E: Ad=81(I) Atr=05(Isoc) MxPS= 800 Ivl=1ms
> > > I: If#= 1 Alt= 6 #EPs= 1 Cls=0e(unk. ) Sub=02 Prot=00 Driver=(none)
> > > E: Ad=81(I) Atr=05(Isoc) MxPS= 956 Ivl=1ms
The uvc driver status is obviously an issue as we’re on a 2.4 Kernel. No uvc support on 2.4.
So…, we either compile 2.6 with UVC support, and reflash, or we continue to use Mavericks driver in lieu of any source.
With that, you have some background on things..
Now, why does this cause a reboot on some machines?
Well, the hardware is different, so the hardware isn’t seen by the driver.
From looking at a few different boards I have seen a few devices id’s used.
So far I have seen these:
vid_0c45/pid_62f1 – (The Sonix Chipset allows you to write pid’s into the device, so this can be changed, or alternately change the driver from 62c0 to 62f1 on these models)
vid_0c45/pid_62c0 – (Our driver is compiled for this)
For those of you with rebooting machines, remove ‘camera &’ from the boot sequence, recompile the kernel with USB verbose debug message logging, and start posting your vid/pid’s here so we can compare, and add to the list.
If someone twisted my arm I could probably oblige…
References:
http://www.beyondlogic.org/uClinux/bflt.htm – bFLT file format details.
http://www.garykessler.net/library/file_sigs.html – Common file format headers
http://www.openwiz.org/wiki/BWFWTools_Release – bFLT unzip and other tools
http://www.hex-rays.com – IDA Pro and Hex-Ray ARM Decompiler
http://www.sigmaplayer.com/filebase.php?d=1&id=13&c_old=5&what=c&page=1 – Arm Decompiler
http://www.ideasonboard.org/uvc/
http://read.pudn.com/downloads127/sourcecode/unix_linux/539050/zc030x/zc030x_cameras.c__.htm
http://www.hackchina.com/r/54654/zc030x_i2c.c__html
http://linux.downloadatoz.com/linux-kernel-webcams-driver-gspca-spca5xx/
http://ww2.cs.fsu.edu/~rosentha/linux/2.6.26.5/docs/video4linux/sn9c102.txt
Files from this post:
(arm2html.exe, bFLT gunzip perl script, original camera bFLT, uncompressed camera bFLT, and camera asm source )
Camera Disassembly, unpacked bFLT and Tools
As I’ve been doing a bit more hardware stuff recently, I thought I’d get some more tools.
Yes, Taobao is wonderful 🙂
I already have a scope – the eminently hackable Rigol DS1052e 50mhz (now running at 100mhz cough).
I already have a bench PSU.
I also have a JTAG device (although its a chinese clone, but hey, it works).
… and I also have a whole bunch of ttl -> rs232 adaptors (as they fry when you’re not careful).
I also have a 3d printer 🙂
I also have a few Arduino bits and bobs, as well as some much more capable ARM dev kits (which I prefer).
What more could I want?
Well, a bus-pirate – but Seeeeeeeeeeed studio still haven’t gotten their sh*t together, and I can’t seem to order from them.
Grumble. Tried again today, but my credit card is rejected as usual. Works on every other site, but theirs..
Still, whilst checking out Seeed Studio’s blog I noted that they mentioned Shanghai (finally!).
Apparently, there’s a hardware hacker dev lab *right* by where I used to live.
They’re up on 50 Yong Jia Lu / website is – http://xinchejian.com/.
I still can’t believe I hadn’t heard about them.
Also slightly annoyed that I missed the last few open days they had.
Going to have to go visit after CNY, and make some friends there 🙂
Lawrence.
The new year has arrived, and so has cheaper pricing for internet access at home.
Currently for home use, there are 3 options for internet over fixed lines.
- China Telecom with ADSL over Copper / FTTB+Lan / FTTH
- China Unicom
- Orient Cable
Incumbent China Telecom has reduced their rates significantly since last year, as Unicom is encroaching on their space.
Lets start with Unicom’s packages:
The latest pricing for 2011 for Unicom is here:
http://www.sh.chinaunicom.com/family/ywcp/jtkh/kdl/index.html#lt_dw_md
Unicom can provide up to 30M for home use (assuming your area can have Fibre access via FTTB+Lan or FTTH)
10M is 198 /month
20M is 248 / month
30M is 288 / month
If you don’t have fibre, and can only get ADSL lines, then your max speeds will be 4M – 6M depending on distance from the local substation.
4M is 144 / month
6M is 168 / month.
Shanghai Telecom has committed to rolling out Fibre to all users by 2013 though, so most areas will start to see fibre availability coming soon.
Shanghai Telecom
Shanghai Telecom pricing is obviously being directly targeted by Unicom’s. Each Unicom price point has been aimed squarely at beating Telecom’s.. Competition is good, although Unicom could do better. Shanghai Telecom has far better backend infrastructure though, and that’s going to take time for Unicom to improve on.
Shanghai Telecom’s current best value package is this:
http://sh.ct10000.com/pptc/ehome/bundle/e8/gwe8sqb/
This offers 10M internet (again with the caveat of Fibre availability in your area) for the sum of 188RMB a month. If you don’t have Fibre, they reduce that price by 10RMB and provide the standard 2M / month (which in comparison with standard adsl rates is not a good deal).
They sell this as a bundle with Telephone access also, so you get reduced phone rates too.
This includes a few other random things like incoming caller ID and custom ringtones for your callers, as well as 30hrs of monthly wifi access assuming you need to use or can find their wifi when around town.
Bundles are:
E8 – ( http://sh.ct10000.com/pptc/ehome/bundle/e8/gwe8sqb/ )
10M – 188rmb /month.
E9 (http://sh.ct10000.com/pptc/ehome/bundle/e9/zxe92011/)
20M – 369rmb / month
Orient Cable
Last and least, we have Orient Cable. OCN rates are here
http://www.ocn.net.cn/gsgg_cuxiao07.html
I’m not going to bother listing their package in detail, as its pretty crappy.
1M for 110 / month.
Looking at the options, my pick for best choices would be
Fibre users:
China Telecom at 188rmb / month with 10M if you can get it value for money wise.
For the speed demons China Unicom for 30M / month.
ADSL only:
China Unicom for their 4M or 6M options if you can only get ADSL installed in your area.
Unicom’s may not offer this in your area though, so you may be stuck on the standard 2M for 150/month till they upgrade lines.
Good luck!
China Mobile may start coming to their senses at some point and offer unlimited fixed wireless, but for now their offerings are too expensive for home use.
Lawrence.
One or two of our servers have been a little bit overloaded recently.
They’re going to be replaced with beefier machines, but due to a number of issues I haven’t been able to replace them yet.
Issue #1 – Pre expo, we weren’t allowed to replace anything.
Issue #2 – Post Expo, I’m no longer allowed in the data center!
We’re working on sorting issue #2 out, but in the interim I need to keep the older machines running.
I was previously using Monit to monitor system load.
Monit would be a good solution – it has a web ui, it can stop services if system load goes too high, and generally works when everything else is failed. This is great when things go poopy, but it has one fatal issue.
It doesn’t know how to restart stuff if load is back to normal.
This typically means that something will put the server load into unusability for a sustained period of time (due to lots of visitors), monit will go ooh, apache has gone awol, and stop it.
Unfortunately if its back to normal, monit doesn’t have a way to start it up again, so I need to manually go to the monit page, and start the service. I do get emailed on things like this, but it leads to complaints from the 2 clients that appear to monitor their particular websites more than monit does.
So, I’ve been looking at other solutions.
One such solution is sysfence
While sysfence is severly underdocumented, the examples provided don’t even work!, and appears to be abandoned, it does do the job.
Sysfence is a no bells and no whistles precursor to monit, but it has that killer feature that monit is missing.
So, how do we use sysfence?
apt-get install sysfence
Will install it, but unfortunately no config is installed.
So, start off by creating a /etc/sysfence folder
mkdir /etc/sysfence
cd /etc/sysfence
We’ll need to create a config file for it, so
pico sysfence.conf
My sample sysfence script is below (explanation underneath script)
rule "ApacheStop" {
la1 >= 10.00 or la5 >= 6.0
}
run '/etc/init.d/apache2 stop;'
rule "ApacheStart" {
la1 <=2
}
run once '/etc/init.d/apache2 start;'
rule "warning" { la1 >= 8.00 } run once 'echo "Load High: BACKUP" | mail lawrence@computersolutions.cn'
I'm having issues with apache causing load to rocket, so I've setup some rules as follows:
If load average for 1 minute > 10 (ie server is going bonkers), and load average for the last 5 minutes > 6 then stop apache.
if load average for 1 minute > 8 send me an email.
if load average for 1 minute < 2 then start apache. This will only run one time if load is below 2.
The documentation http://sysfence.sourceforge.net/ goes over how to write a rule. Note that the examples are broken;
eg
if {
la1 >= 8.00
} run once 'echo "SHOW FULL PROCESSLIST" | mysql | mail my@email.com'
Issue? All rules need to have a "rule name" specified.
So a corrected working version would be:
if "some rule" {
la1 >= 8.00
} run once 'echo "SHOW FULL PROCESSLIST" | mysql | mail my@email.com'
Back to our setup..
Now we've setup a ruleset, we need to run it. Calling sysfence /etc/sysfence.conf
Will run it as a daemon.
ps -ef
shows our rulesets running:
root 7260 1 0 05:51 ? 00:00:01 sffetch
root 7261 7260 0 05:51 ? 00:00:00 sfwatch 'warning'
root 7262 7260 0 05:51 ? 00:00:00 sfwatch 'ApacheStop'
root 7263 7260 0 05:51 ? 00:00:00 sfwatch 'ApacheStart'
sffetch is the daemon, and sfwatch are the rules it runs.
As sysfence is quite rudimentary, you'll need to kill it if you change rules.
You'll also need to add it to your startup scripts or create one. I'll be lazy and not go over that right now. If people are interested, add a comment, and I'll put something up.
Sysfence can be downloaded here - http://sysfence.sourceforge.net/ (or via apt-get if on a Debian based OS)
Man page for sysfence below (note examples require adding "rulename" after if... {
or rule ... {
):
NAME
sysfence - system resources guard for Linux
SYNOPSIS
sysfence
<configuration file> [<configuration file> ...]
DESCRIPTION
Sysfence is a resource monitoring tool designed for Linux machines.
While running as daemon it checks resource levels and makes desired
action if some values exceed safety limits.
Sysfence can be used for notifying system administrators when something
goes wrong, stopping services when system performance is dropping too
low and starting them when it's going up again, periodically restarting
memory-leaking processes, dumping system statistics in critical situations.
Sysfence can monitor following resource levels: load average, used and
free memory amount, used and free swap space.
USAGE
Sysfence reads it's configuration from file(s) specified in argument
list. Config files may contain one or more rules describing conditions
and actions to be performed.
Rule has syntax like this:
if {
resource1 > limit1
or
{ resource2 < limit2 and resource3 < limit3 }
}
run once 'command-to-be-run'
The block enclosed within {} brackets describes condition. When it's
result is TRUE, following command is invoked.
The once keyword is optional. If present, the command is executed only
once after condition becomes TRUE. Next execution will take place only
if condition becomes FALSE and then TRUE again. Without once keyword,
command is invoked periodically, after every resource check that gives
TRUE, no matter what was the condition result before.
Command specified right after run keyword is passed to /bin/sh, so it
may contain more than one instruction or even whole script. But be
careful - rule checking is suspended unless command execution has been
completed! (Other rules are unaffected.)
As resources, following ones can be given:
- la1
- - load average during last minute.
- la5
- - load average during last 5 minutes.
- la15
- - load average during last 15 minutes.
- memfree
- - lower limit for free memory amount.
- memused
- - upper limit for memory used by processes.
- swapfree
- - lower limit for free swap space.
- swapused
- - upper limit for swap space in use.
EXAMPLES
Do you have problems with MySQL server choking and freezing whole
system? I do. To find queries that cause problems, you may use:
if {
la1 >= 8.00
} run once 'echo "SHOW FULL PROCESSLIST" | mysql | mail my@email.com'
Of course, that wouldn't prevent your system from being blocked, but
following rule could. MySQL will be restarted if LA for last minute
is over 10.0 or LA for last five minutes is over 6.0.
if { la1 >= 10.00 or la5 >= 6.0 }
run '/etc/rc.d/init.d/mysql stop; sleep 120; /etc/rc.d/init.d/mysql
start'
We may also restart some services that probably have memory leaks and
use lots of swap space if not restarted periodically. Let's assume
that 256MB of used swap is enough to give our Zope server a break.
if {
swapused >= 256M
} run '/etc/rc.d/init.d/zope restart'
We may also alert admins... Notice that you don't have to be r00t:
if {
la15 > 4.0
and
{
swapfree < 64M
or
memfree < 128M
}
} run 'echo "i wish you were here..." | sendsms +48ADMINCELLPHONE'
Using sysfence version 0.7 or later you may give rule a name that will
be used in logs:
rule "high load" { la1 > 3.0 and la15 > 2.0 } log
rule keyword has the same meaning as if. There are also synonymes for
other keywords. Detailed list is included within sysfence package.
You can find an example config file in /usr/share/doc/sysfence/example.conf.
AUTHOR
Sysfence was written by Michal Saban (emes at pld-linux org) and
Mirek Kopertowski (m.kopertowski at post pl)
This manual page was created by Lukasz Jachowicz <honey@debian.org>,
for the Debian project (but may be used by others). It is based on
the http://sysfence.sf.net/ page.
Comment spam is a big load on servers – recently troubleshooting some intermittent load issues on one of our webservers, I discovered that one page was getting hammered by comment spam abuse – the page had already generated over 400,000 comments, which was causing the server to be slightly unhappy loadwise.
After clearing out the comment table for that clients site, I looked into solutions.
mod_security was an obvious one, and the atomic corp rules seem to be better than the default mod_security ones (which break most popular apps, sigh). However, although the gotroot rules were good at blocking comment spam, they don’t block the ip’s, so persistent spammer bots will still hammer the server.
So, whats a solution?
Blocking the ip’s dynamically for a short period.
Assuming you have a standard setup, the below should be of interest.
I stick my ruleset into /etc/modsecurity2, so amend your url’s accordingly.
My default initialise code is below (pretty much stock except for the end part)
/etc/modsecurity2/modsecurity_crs_10_config.conf
SecRuleEngine On
SecRequestBodyAccess On
SecResponseBodyAccess On
SecResponseBodyMimeType (null) text/html text/plain text/xml
SecResponseBodyLimit 2621440
SecServerSignature Apache
SecComponentSignature 200911012341
SecUploadDir /var/asl/data/suspicious
SecUploadKeepFiles Off
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogType Concurrent
SecAuditLog /var/log/apache2/modsec_audit.log
SecAuditLogParts ABIFHZ
SecArgumentSeparator "&"
SecCookieFormat 0
SecRequestBodyInMemoryLimit 131072
SecDataDir /var/asl/data/msa
SecTmpDir /tmp
SecAuditLogStorageDir /var/asl/data/audit
SecResponseBodyLimitAction ProcessPartial
SecPcreMatchLimit 100000
SecPcreMatchLimitRecursion 100000
SecDebugLog /var/log/apache2/modsec_debug.log
SecDebugLogLevel 3
### Ruleset additions for blocking ########################################
# Make sure to clear the default action
SecDefaultAction phase:1,pass
# Initialize collection and deprecate by 3 points per day (86400 seconds)
SecAction phase:1,initcol:IP=%{REMOTE_ADDR},deprecatevar:IP.spam=3/86400,nolog
# If there are already >15 spam points for this IP, then drop
# the connection and add 1 point (instead of 3, as below).
SecRule IP:spam "@gt 15" phase:1,setvar:IP.spam=+1,drop,setenv:spam=spam
# Clear the default action for any mod_security rules later in httpd.conf.
SecDefaultAction phase:1,pass
### End Ruleset additions for blocking ####################################
What does it do? Basically we keep score of any ip address that visits the site.
Every day any scores depreciate by 3 points per ip.
If any ip scores more than 15 points, we block it.
Now, as it stands, its pretty useless, as we don’t do any scoring.
So, next we’ll need to add some scoring to some of the rules.
In our case, we were getting seriously hammered by pharmacy spam (mostly out of latvia)
So, I went into the default gotroot ruleset that I installed into /etc/modsecurity2
Comment spam is in this file: 30_asl_antispam.conf
I added some scoring there.
In my case, we saw hundreds of thousands of attempts for levitra and tramadol spam, so I went to the pharmacy scoring, and added this:
setvar:IP.spam=+15
in the SecRule check.
This means that any positive lookup for that rule will give the ip a 15 point score.
That will block that attacker ip from the site for 5 days (as we decrement 3 points per day for a given ip).
The setvar needs to go after the “capture,…” part of the rule.
eg
(don’t copy, paste this, you’ll need to add “setvar:IP.spam=+15” into your existing file)
# Rule 300040:
SecRule ARGS|!ARGS:/domain/|!ARGS:description|!ARGS:redirect_to|!ARGS:setting[banemail]|!ARGS:/username/|!ARGS:/user_name/|!ARGS:/page_content/|!ARGS:/search/|!ARGS:/email/!ARGS:Mensaje|!ARGS:/product/|!ARGS:/domain/|!ARGS:description$
"capture,setvar:IP.spam=+15,id:300040,rev:7,severity:2,msg:'Atomicorp.com - FREE UNSUPPORTED DELAYED FEED - WAF Rules: Spam: Pharmacy',logdata:'%{TX.0}'"
I added this to appropriate places, and watched for our first victim^Mspammer in our site logs.
Bingo – here we have a spammer trying to add to a url on one of our sites:
[23/Dec/2010:11:39:45 +0800] [www.chou.cn/sid#bcf3f338][rid#b339b670][/gallery/papratiti/img_5993.jpg.php][1] Access denied with code 403 (phase 2). Pattern match “(?:buy[-_ ]?(cheap)?[-_ ]?(?:adipex|suboxone|pseudovent|topamax|trazodone|prevacid|zyrtec|xenical|toprol|zoloft…” at ARGS:comment. [file “/etc/modsecurity2/30_asl_antispam.conf”] [line “194”] [id “300061”] [rev “10”] [msg “Atomicorp.com – FREE UNSUPPORTED DELAYED FEED – WAF Rules: Spam: Pharmacy”] [data “xanax”] [severity “CRITICAL”]
[23/Dec/2010:11:43:16 +0800] [www.chou.cn/sid#bcf3f338][rid#bfac9d68][/gallery/papratiti/img_5993.jpg.php][1] Access denied with connection close (phase 1). Operator GT matched 15 at IP:spam. [file “/etc/modsecurity2/modsecurity_crs_10_config.conf”] [line “40”]
Bam, that ip is blocked (for 5 days). Next time around, as its the first rule to run, it will block immediately without processing other rules.
You can amend the block times by decreasing the increments or scoring as per your requirements.
All in all, this is an easy amendment to make to the gotroot rules, and makes for faster experience for users.
Thanks to http://linux.icydog.net/apache/commentspam.php for his implementation, this is pretty much based off that.
Lawrence
First my tale of woe:
I just got burgled back home, and they took two of my laptops. One was a small hackintosh Mini9, which I’m not too worried about, but the other was a top of the line IBM Thinkpad which I’d just donated it to my cousin for his studies.
Sadly we didn’t have insurance, and the likelyhood of getting it back is close to zero. I will be watching Gumtree closely though for the next few weeks!
As anyone who owns a laptop can tell you, the most worrying things to worry about are:
The dreaded coffee spill
(only 1 client this month – get those keyboard… er… condoms!)
The whoops I dropped it, now my hard drive is dead / screen is broken
(Too many to mention, you do have backups right?)
and the most dreaded of all…
The oh @#$! I left it in a Taxi. Or if you live in South Africa – someone affirmative actioned it..
While #1 and #2 are relatively easy to recover from (albeit costly), #3 does pose issues.
There are solutions though.
One such solution is one I’d suggested to my brother to install prior to the theft – install Prey. Unfortunately for me he hadn’t done it, doh!
Prey (http://preyproject.com/) is an elegant solution to hardware protection.
Its an install and forget free service that tracks up to 3 computers (more on a paid basis).
How does it work?
You install the Prey software on your computer, and signup for an account (all part of the install).
Their software then silently runs in the background. There are zero options to set, just install, and setup a user account.
Its really set, and forget.
Nothing happens until the computer goes missing – then you can choose from a variety of different useful actions, ranging from taking a photo of the person using the computer (assuming like most laptops these days it has a built in webcam), to monitoring what they’re doing by taking snapshots of the current screen. This is all done silently without the thief knowing that its going on.
This is great for catching them on Facebook, uh Kaixin001 and spotting their user name.
Below is a view of the logged in screen for my current laptop
The modules page shows the list of items I can turn on when needed.
The useful ones will be the webcam, and session photo captures. Geolocation is a crapshoot in China, and having the IP info is pretty useless here unless you know someone at China Telecom.
Hopefullly this will be one piece of software that I never use 🙂
Highly recommended for those with expensive equipment that moves around.
Dear Clients,
The government has imposed extended legislation regarding domains and domain hosting in China. As part of these new requirements, we will be required to keep and maintain a set of registration documents for each domain we host.
We will also need to impose a small service fee (300rmb per client for first domain, 100rmb for subsequent domains) for providing assistance with application submission, so that we can cover our costs.
We are now required to do the following for all .CN domains we administer according to Chinese Law.
- Take a color headshot of the contact person of the Applicant Company.This photo must be taken in our office against an official backdrop image.
- Provide:
– A copy of the Certificate of Business License of Legal Entity for the applicant company or a copy of the Certificate of National Organization Code of the applicant company.– A copy of the Chinese Resident Identity Card of the contact person of the applicant company.
Applicants will need to bring the originals to our office so that we can scan them in color in an acceptable format for CNNIC and MII.
- Have the applicant sign/ chop a registration form confirming all information is correct.
- Ensure that your ICP 备案 is up to date and information is correct.
- Verify domain content, and ICP presence on your site.
Note that no personal .CN domain registrations are currently allowed for foreigners.
We are required to submit a valid China business licence, and Chinese ID to the applicable authorities.
If this information cannot be submitted, and your domain url ends in .CN , you will lose your .CN domain..
This information has to be submitted by us to the relevant involved bureau’s (MII, CNNIC, Shanghai Telecom) before the end of October.
We appreciate that this is quite short notice, and urge you to arrange a time to come to our office to fulfil these requirements before the end of October.
We will be updating our ICP and other customer support sites shortly to take into account new requirements.
Mini FAQ
What is a .cn domain?
Any domain that ends in .cn
eg www.computersolutions.cn
www.computersolutions.com.cn
Is this applicable to .com or other domains too?
Yes.
We are required to submit and verify identification information for all domains that we host prior to November 1st.
All clients with domains will need to submit information by coming to our offices with the required information.
Where can I read more about this?
http://www.bakermckenzie.com/RRDomainNameWebsiteRegistration/
http://www.aplf.org/new-regulations-for-registering-domain-names-in-china/
(Note that requirements were extended on October 1st to be applicable for all domains, not just new registrations.)
——-
尊敬的客户,
中国政府发布了关于域名和域名托管的扩展规定。
根据这些最新规定,我们需要为每个托管的域名保留维持一些注册资料。
具体来说,我们需要完成以下所有事务。根据中国法律规定我们管理CN网域。
采集申请公司联系人的彩照一张,照片必须在我们公司使用正式背景图像采集。
-为申请公司提供一份公司法人的营业执照证书复印件或者提供一份申请公司的全国组织结构代码证书复印件。
– 提供一份申请公司联系人的中国居民身份证复印件一份。
-你需带上原始件,以便我们能够彩色扫描为CNNIC 和 MII格式。
申请人需签名确认所有信息的正确性。
请确保ICP备案更新及时,信息准确。
核实网站域名内容和ICP内容。
请注意目前外国客户不允许注册个人CN域名。
我们需向申请局提交合法的中国营业执照和中国居民身份证。
如果此信息不能提交,那么你将失去 CN域名。
该信息需在十月底前由我们提交给相关部门(信息产业部,互联网络信息中心,上海电信)。
我们发布这则简短通知,希望你们安排时间在十月底前来我们办公完成这些要求。
考虑到新规定,我们将会持续更新ICP和其他客户支持站点。
Mini FAQ
什么是cn域名?
任何以.cn结尾的域名
如www.computersolutions.cn
www.computersolutions.com.cn
这个对.com 或其他域名也适用吗?
适用
我们需在11月1日前提交并核实所有托管域名的确认信息。
域名客户需携带所需信息来我们办公室提交。
在哪里能获得更多信息呢?
http://www.bakermckenzie.com/RRDomainNameWebsiteRegistration/
http://www.aplf.org/new-regulations-for-registering-domain-names-in-china/
请注意10月1日新增的要求对所有域名都适用,不只是新注册域名。
Archives
- November 2019
- October 2019
- August 2019
- April 2019
- February 2017
- September 2016
- June 2016
- May 2016
- September 2015
- August 2015
- June 2015
- April 2015
- December 2014
- October 2014
- September 2014
- July 2014
- June 2014
- April 2014
- October 2013
- July 2013
- May 2013
- April 2013
- March 2013
- January 2013
- December 2012
- October 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012
- March 2012
- December 2011
- November 2011
- October 2011
- September 2011
- July 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- May 2009
- April 2009
- March 2009
- February 2009
- January 2009
- December 2008
- November 2008
- October 2008
- September 2008
Categories
- Apple
- Arcade Machines
- Badges
- BMW
- China Related
- Cool Hunting
- Exploits
- Firmware
- Food
- General Talk
- government
- IP Cam
- iPhone
- Lasers
- legislation
- MODx
- MySQL
- notice
- qmail
- requirements
- Reviews
- Service Issues
- Tao Bao
- Technical Mumbo Jumbo
- Things that will get me censored
- Travel
- Uncategorized
- Useful Info