小米 sells a small little wifi camera which works quite painlessly.
I’ve bought several over the last few years.

My recent ones don’t work oversea’s though (which is where I install them).
Its extremely bogus selling equipment that is limited to a country, especially when they don’t tell you about it.

Annoying, as my older models aren’t “region locked”, but the newer ones are.

There is a fix for it, although it will likely get more difficult to patch at xiaomi’s whim.
If so, I’ll probably stop buying the camera’s. Lovely how vendors want to mess with customers…

—

First up, enable telnet on the camera.
Get an SD card or take the card out of the camera.

Create a folder named test on the card.
Create a plain text file called equip_test.sh in that folder, and add the following bash script:


#!/bin/sh
# Telnet
if [ ! -f "/etc/init.d/S88telnet" ]; then
echo "#!/bin/sh" > /etc/init.d/S88telnet
echo "telnetd &" >> /etc/init.d/S88telnet
chmod 755 /etc/init.d/S88telnet
fi
dr=`dirname $0`
# fix bootcycle
mv $dr/equip_test.sh $dr/equip_test.sh.moved
reboot

The script will enable telnet on the camera, and then rename the script so it doesn’t run again on the next boot.

Stick the prepared card into the camera, power on, and it should reboot (twice).
If you check the open ports on the camera ip you should now see port 23 (telnet) is open.

Login with the default user/pass (as below) via telnet

User: root
Password: 1234qwer


Once telnet’d in, enter the following, line by line –

Find and kill the watchdog process

killall watch_process


Find and kill /home/cloud process so we can edit it without the watchdog watch_process restarting it

killall cloud


Change the check within /home/cloud executable to query a fake domain so it never returns a failure, then reboot.


sed -i 's|api.xiaoyi.com/v4/ipc/check_did|api.xiaoyi.foo/v4/ipc/check_did|g' /home/cloud
reboot


Notes – The camera checks for the country by running an executable called /home/cloud
This calls an api at api.xiaoyi.com which does a ip check, then returns a true or false value if it thinks you’re in China or not. We change the /home/cloud executable ip check call to a bad url, which means it can’t perform its check.

Worked on my camera’s running version 1.8.6.1Q_201607271501

The SSL certificate for the all servers have been updated to use a wildcard certificate.

We *finally* changed over to use a wildcard cert, as pricing has come down enough to not warrant having separate certificates per server.
Our new wildcard certificate is valid until 2019.

What does this mean for you?

The bad news
Really old browsers won’t be able to open our site
If you are an XP user running IE6, you won’t be able to load our encrypted sites anymore. We strongly suggest you upgrade though if you fall into that category!
Same goes for those running Android 2.x (which is equally ancient in computer terms).

The good news
Email is now encrypted point to point using AES256 SHA encryption where possible, and webmail is SHA256 encrypted from server to your browser.
Mail servers that support it (i.e. all of ours, plus the major providers like Google, Yahoo etc, will send encrypted mail to our servers).
Mail Headers will include things like the below if encryption is supported –
Received: from usa4.computersolutions.cn (162.210.36.26) by mail.computersolutions.cn with AES256-SHA encrypted SMTP;

Lastly – our new cert gets us a test rating of A at the SSL Labs site.
https://www.ssllabs.com/ssltest/analyze.html?d=computersolutions.cn&latest

Some of our clients are experiencing delivery issues to some domains that use Gmail/Google for their email.

I previously covered that here – http://www.computersolutions.cn/blog/2015/04/gmail-and-other-google-hosted-mail-delivery-issues/

The issue is that China is still blocking Gmail/ Google hosted mail, and the recipient domain hasn’t setup their MX records correctly.

This is fine for servers outside of China, where all of googles mail servers (should) work, but breaks things for those inside China, where only a few servers are reachable.

Google hosted mail settings are here: https://support.google.com/a/answer/33915?hl=en

You’ll note that there are 5 different email servers that are listed in priority order.

Priority Mail Server
1 ASPMX.L.GOOGLE.COM.
5 ALT1.ASPMX.L.GOOGLE.COM.
5 ALT2.ASPMX.L.GOOGLE.COM.
10 ALT3.ASPMX.L.GOOGLE.COM.
10 ALT4.ASPMX.L.GOOGLE.COM.

For mail servers, the higher number is more important, so a priority of 1 will be the first server tried, then the next highest number, and so on.

If I try to connect to the servers from China.

telnet ASPMX.L.GOOGLE.COM 25
Trying 74.125.200.27…
(times out)

telnet ALT1.ASPMX.L.GOOGLE.COM 25
Trying 173.194.72.26…
(times out)

telnet ALT2.ASPMX.L.GOOGLE.COM 25
Trying 74.125.25.26…
(times out)

telnet ALT3.ASPMX.L.GOOGLE.COM 25
Trying 64.233.169.26…
Connected to ALT3.ASPMX.L.GOOGLE.COM.
Escape character is ‘^]’.
(yay, we have a winner!)

telnet ALT4.ASPMX.L.GOOGLE.COM 25
Trying 74.125.70.27…
Connected to ALT4.ASPMX.L.GOOGLE.COM.
Escape character is ‘^]’.
(yay, we have a winner!)

So, we can see that alt3, alt4 work, but none of the others do (as of 9th September 2015 from Shanghai)

So, some rudimentary testing shows that some servers work, and some do not.
How does that apply to real world examples.

Lets look at a non-working domain – ihg.com

dig mx ihg.com

;; ANSWER SECTION:
ihg.com. 600 IN MX 100 aspmx3.googlemail.com.
ihg.com. 600 IN MX 50 alt1.aspmx.l.google.com.
ihg.com. 600 IN MX 50 alt2.aspmx.l.google.com.
ihg.com. 600 IN MX 100 aspmx2.googlemail.com.
ihg.com. 600 IN MX 10 aspmx.l.google.com.

You should easily be able to see 2 things.
1 – that the MX records are not as per Google settings.
2 – that the 2 working MX records are not listed.

This means that while their MX records probably work oversea’s, they will not be deliverable from China. They need to amend their MX records to Googles recommended settings.

Lets look at another example.

dig mx rsms-west.com

;; ANSWER SECTION:
rsms-west.com. 6238 IN MX 30 alt2.aspmx.l.google.com.
rsms-west.com. 6238 IN MX 10 aspmx.l.google.com.
rsms-west.com. 6238 IN MX 40 aspmx2.googlemail.com.
rsms-west.com. 6238 IN MX 50 aspmx3.googlemail.com.
rsms-west.com. 6238 IN MX 20 alt1.aspmx.l.google.com.

Once again, we can see that the alt3, and alt4 servers are missing, and unfortunately none of the other listed servers are connectable from China.

Lastly, lets look at a working server

dig mx teamsequel.com

teamsequel.com. 12878 IN MX 1 ASPMX.L.GOOGLE.com.
teamsequel.com. 12878 IN MX 5 ALT1.ASPMX.L.GOOGLE.com.
teamsequel.com. 12878 IN MX 5 ALT2.ASPMX.L.GOOGLE.com.
teamsequel.com. 12878 IN MX 10 ALT3.ASPMX.L.GOOGLE.com.
teamsequel.com. 12878 IN MX 10 ALT4.ASPMX.L.GOOGLE.com.

You can see that they have the correct Gmail settings as per Gmail / Google settings page, and mail to them is deliverable (as alt3, alt4 are currently not being blocked by the beneficent government of China).

Unfortunately as this is an issue that is out of our control (MX records are incorrect, and China is being difficult), we cannot mitigate against it. The affected domains will need to amend their MX records appropriately as per the page here- https://support.google.com/a/answer/33915?hl=en.

The Mercury MAC1200R (TPLINK WR6300) is one of the cheapest AC based routers on the planet, as I have a few AC devices I thought I’d buy one to try out. They’re available for under RMB120 or so online.

Mercury hardware is TPLink hardware, albeit using a different name. Same stuff, same factory, different casing.

The router has 64M ram, and 8M flash. CPU / Wifi AR9344 / QCA9882

OpenWRT says it has support, so I had a quick go at taking a look at flashing one.

First steps:

Open up the unit (remove the 2 screws underneath, then use a spare credit card or similar to jam open the plastic case).

The serial headers are to the right of the SoC.
You’ll need a rs232 to ttl or usb to ttl adaptor.

Pin 1 is GND
Pin 2 is TX (connect to rx)
Pin 3 is RX (connect to tx)
Pin 4 is VCC (don’t connect).

Port speeds are 115200,8,n,1

Connect up, and you should see a flurry of activity on boot.

To get into the bootloader (uBoot), you’ll need to type tpl
May take a few tries, so prepare to pull power and retry again, and again till you get in.

You’ll need a TFTPD server running on 192.168.1.100 to push files over. My Mac no haz ethernet (and Thunderbolt to Ethernet is crud), so I use my venerable X40 and tftpd software.

Firmware is up here. (Or you can compile your own) -> http://pan.baidu.com/s/1hqkVOfa

Firmware locations in flash (see below)
printenv
bootargs=console=ttyS0,115200 root=31:02 rootfstype=jffs2 init=/sbin/init mtdparts=ath-nor0:256k(u-boot),64k(u-boot-env),6336k(rootfs),1408k(uImage),64k(mib0),64k(ART)
bootcmd=bootm 0x9f020000
bootdelay=1
baudrate=115200
ethaddr=0xba:0xbe:0xfa:0xce:0x08:0x41
ipaddr=192.168.1.111
serverip=192.168.1.100
dir=
lu=tftp 0x80060000 ${dir}u-boot.bin&&erase 0x9f000000 +$filesize&&cp.b $fileaddr 0x9f000000 $filesize
lf=tftp 0x80060000 ${dir}db12x${bc}-jffs2&&erase 0x9f050000 +0x630000&&cp.b $fileaddr 0x9f050000 $filesize
lk=tftp 0x80060000 ${dir}vmlinux${bc}.lzma.uImage&&erase 0x9f680000 +$filesize&&cp.b $fileaddr 0x9f680000 $filesize
stdin=serial
stdout=serial
stderr=serial
ethact=eth0

Environment size: 686/65532 bytes

Upload to the uBoot

It should boot into OpenWRT, you can flash from there.


...[long kernel bootlogs redacted]

I highly recommend replacing the bootloader with an unlocked one.
i.e. this one – https://github.com/pepe2k/u-boot_mod

Bootloader sits at 0x9f000000 and is 64kb in size.
Suggest load in ram at 0x80060000 like they do, and flash from there.

eg from the bootloader –

tfpboot 0x80060000 uboot.bin
erase 0x9f000000 +0x20000
cp.b 0x80060000 0x9f000000 0x20000


Pray that it didn’t mess up (or you need an spi flasher…), and reboot!

Other bits n bobs –

You’ll need to adjust the opkg.conf file

ssh into the router @ 192.168.1.1 / root / root

mv /etc/opkg.conf /etc/opkg.conf.orig

#src/gz barrier_breaker_base http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages/base
#src/gz barrier_breaker_management http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages/management
#src/gz barrier_breaker_oldpackages http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages/oldpackages
#src/gz barrier_breaker_packages http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages/packages
#src/gz barrier_breaker_routing http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages/routing
#src/gz barrier_breaker_telephony http://downloads.openwrt.org/snapshots/trunk/ar71xx/packages/telephony' >> /etc/opkg.conf

…and install Luci (the openwrt webui)

opkg update
opkg install luci


You should be able to visit http://192.168.1.1 and start changing settings.

Yet another SSL vulnerability has hit the news – the Poodle SSLv3 vulnerability.

Our servers are already patched against this (we’ve disabled SSLv2 and SSLv3 functionality, and use TLS).

You can check this on the 3rd party site here –

https://www.ssllabs.com/ssltest/analyze.html?d=mail.computersolutions.cn&s=211.144.68.16

Unfortunately this now means that Windows XP and IE6 are no longer supported.

Our rating from the SSLLabs checker is below.
Note that the A- rating is due to our certificate, not our security!

(We can only update that in 2016 when it comes up for renewal).

SSL negotiation in use on our server

In January, I upgraded to 100M fibre, and paid upfront for the year (RMB2800).

While I was on vacation, my FTTB at home stopped working, so we called Shanghai Telecom.

What had actually happened was that there was a screwup with the account setup, and they’d put me on a monthly bill *and* 100M.
After 6 months, they decided that I hadn’t paid my bill, and cancelled my 100M fibre account!
Staff eventually sorted it out, and Telecom gave us a 6 month credit.
Even so, I ended up coming back to a crappy E8 wifi + modem setup and my router set to use DHCP.

The Shanghai Telecom unit was setup for a maximum of 16 wifi devices, and uPNP was disabled, sigh.

I prefer to use my own equipment, as I generally don’t gimp it, so I called Telecom to ask for my “new” account details so I could replace it.

Unfortunately the technician had changed the password, and the 10000 hotline didn’t have the new pass, or the LOID.

I called the install technician who’d installed it in my absence, but he wasn’t very helpful, and told me I couldn’t have it. Surprise…

What to do.

I took a look at their modem, and thought it should be fairly easy to try get the details from it.

Did a bit of googling, and found that it had an accessible serial port, so opened up the unit, and connected it up.

After a bit of cable fiddling, got a connection @ 115200 / 8n1

Cable pinout should be –
GND | MISSING PIN | TX | RX | VCC

I’ll add some photos later.

With some more fiddling around, I got terminal access (accidentally!) with some prudent Ctrl C/ Ctrl Z’ing during the boot process as something crashed and I got a terminal prompt.
Its vxware, although the boot process does look quite linuxy.

Lots of interesting commands..

 > ls -al
telnetd:error:341.568:processInput:440:unrecognized command ls -al
 > help
?
help
logout
exit
quit
reboot
brctl
cat
loglevel
logdest
virtualserver
ddns
df
dumpcfg
dumpmulticfg
dumpmdm
dumpnvram
meminfo
psp
kill
dumpsysinfo
dnsproxy
syslog
echo
ifconfig
ping
ps
pwd
sntp
sysinfo
tftp
voice
wlctl
showOmciStats
omci
omcipm
dumpOmciVoice
dumpOmciEnet
dumpOmciGem
arp
defaultgateway
dhcpserver
dns
lan
lanhosts
passwd
ppp
restoredefault
psiInvalidateCheck
route
save
swversion
uptime
cfgupdate
swupdate
exitOnIdle
wan
btt
oam
laser
overhead
mcpctl
sendInform
wlanpower
zyims_watchdog
atbp
ctrate
testled
ipversionmode
dumptr69soap
lan2lanmcast
telecomaccount
wanlimit
namechange
userinfo
localservice
tcptimewait
atsh
option125Mode
eponlinkper
setponlinkuptime
loidtimewait
phonetest
 

First up, dump the nvram

> dumpnvram
============NVRAM data============
nvramData.ulVersion=6l
nvramData.szBootline=e=192.168.1.1:ffffff00 h=192.168.1.100 g= r=f f=vmlinux i=bcm963xx_fs_kernel d=1 p=0 c= a= 
nvramData.szBoardId=      XPT2542NUR
nvramData.ulMainTpNum=0l
nvramData.ulPsiSize=64l
nvramData.ulNumMacAddrs=10l
nvramData.ucaBaseMacAddr=??Umo
nvramData.pad=
nvramData.ulCheckSumV4=0l
nvramData.gponSerialNumber=             
nvramData.gponPassword=           
nvramData.cardMode=-1
nvramData.cardNo=  000000000000000000
nvramData.userPasswd=telecomadmin31407623
nvramData.uSerialNumber=32300C4C755116D6F
nvramData.useradminPassword=62pfq
nvramData.wirelessPassword=3yyv3kum
nvramData.wirelessSSID=ChinaNet-WmqQ
nvramData.conntrack_multiple_rate=0
============NVRAM data============

Nice, got the router admin pass already.
– nvramData.userPasswd=telecomadmin31407623
(user is telecomadmin).

I actually needed the login details, this turned out to be via

 > dumpmdm 

This dumped a rather large xml style file with some interesting bits

[excerpted are some of the good bits – the whole file is huge]


FALSE
e8ftp
e8ftp
21
TRUE
FALSE
TRUE
e8telnet
e8telnet
23
FALSE
admin
v2mprt



Hmm, telnet, and a password!
Telnet is not enabled by default, nor is FTP.

It also had the pppoe user/pass which was what I was looking for, and the LOID, which I needed to stick into my modem.
Score.

While that was pretty much all I needed, I decided to enable Telnet and FTP to play around.

Ok, so how do we enable telnet?

 > localservice
usage:
   localservice show: show the current telnet/ftp service status.
   localservice telnet enable/disable: set the telnet service enable or disable.
   localservice telnetAccess enable/disable: allow access telnet in wan side or not.
   localservice ftp enable/disable: set the ftp service enable or disable.
   localservice ftpAccess enable/disable: allow access ftp in wan side or not.

 > localservice telnet enable
 
> localservice show
Current local services status:
Ftp Service: Disable
Ftp Allow Wan Access: No
Telnet Service: Enable
Telnet Allow Wan Access: No
 
> localservice ftp enable

> localservice show
Current local services status:
Ftp Service: Enable
Ftp Allow Wan Access: No
Telnet Service: Enable
Telnet Allow Wan Access: No
 > save
config saved.

reboot the modem, and see if we can login via ethernet

telnet 192.168.1.1
Trying 192.168.1.1...
Connected to broadcom.home.
Escape character is '^]'.
BCM96838 Broadband Router
Login: telecomadmin
Password: 
Login incorrect. Try again.
Login: e8telnet
Password: 
 > 

Cool, so we now have full access to the device.

There also seems to be a remote monitoring system config’d via devacs.edatahome.com, which maps to a Shanghai Telecom ip.

   http://devacs.edatahome.com:9090/ACS-server/ACS
      http://devacs.edatahome.com:9090/ACS-server/ACS
      hgw
      hgwXXXX1563

and something else called itms.

itms
 itmsXXXX5503

I’ve XXX’d out some of the numbers from my own dump, as I suspect its device / login specific.

I got what I needed though, which was admin access to the modem, despite Shanghai Telecom not telling me.

Would really be nice if they just gave you the PPPoE user/pass and LOID, but that would be too easy…

On my modem, the following were the default passwords:

Console Access (via serial port)

User: admin
Pass: v2mprt

Once in console, you can enable Telnet and FTP.

Telnet (not enabled by default)
User: e8telnet
Pass: e8telnet

FTP (not enabled by default)
User: e8ftp
Pass: e8ftp

To show the http password from console (either local, or via telnet).
dumpnvram

url: http://192.168.1.1
http user: telecomadmin
http pass: (as per nvram, mine was telecomadmin31407623 )

Once in you can see all the important bits. Probably easier to grep the xml file from

dumpmdm

Took me about an hour or so to get to that point, I’m running on my own equipment again, and its not gimped. Worth my time!

I’ve noticed a little spate of password attack attempts via Roundcube – a webmail program we use for mail over at https://mail.computersolutions.cn

Roundcube does have captcha plugins available which will mitigate this, but users will complain if they have to type in a captcha to login for mail.

Fail2ban provides an easy solution for this.

Roundcube stores its logs in a logs/errors file.

If I take a look at a sample login failure, it looks something like the example below

[09-Jun-2014 13:43:38 +0800]: IMAP Error: Login failed for admin from 105.236.42.200. Authentication failed. in rcube_imap.php on line 184 (POST /?_task=login&_action=login)

We should be able to use a regex like:
IMAP Error: Login failed for .* from

However fail2ban’s host regex then includes a trailing ., and fail2ban doesn’t recognise the ip.
I eventually came up with the overly complicated regex below, which seems to work:

IMAP Error: Login failed for .* from <HOST>(\. .* in .*?/rcube_imap\.php on line \d+ \(\S+ \S+\))?$

Lets add detection for that into fail2ban.
First up, we need to add roundcube into /etc/fail2ban/jail.conf

[roundcube]
enabled  = false
port     = http,https
filter   = roundcube
action   = iptables-multiport[name=roundcube, port="http,https"]
logpath  = [YOUR PATH TO ROUNDCUBE HERE]/logs/errors
maxretry = 5
findtime = 600
bantime = 3600

Note that we are not enabling the filter yet.

Change [YOUR PATH TO ROUNDCUBE HERE] in the above to your actual roundcube folder
eg /home/roundcube/public_html/logs/errors

Next, we need to create a filter.

Add /etc/fail2ban/filter.d/roundcube.conf

[Definition]
failregex = IMAP Error: Login failed for .* from <HOST>(\. .* in .*?/rcube_imap\.php on line \d+ \(\S+ \S+\))?$

ignoreregex =

Now we have the basics in place, we need to test out our filter.
For that, we use fail2ban-regex.
This accepts 2 (or more) arguments.

fail2ban-regex LOGFILE  FILTER 

For our purposes, we’ll pass it our logfile, and the filter we want to test with.

eg

fail2ban-regex  /home/roundcube/public_html/logs/errors /etc/fail2ban/filter.d/roundcube.conf  |more

If you’ve passed your log file, and it contains hits, you should see something like this:

Running tests
=============

Use regex file : /etc/fail2ban/filter.d/roundcube.conf
Use log file   : /home/www/webmail/public_html/logs/errors


Results
=======

Failregex
|- Regular expressions:
|  [1] IMAP Error: Login failed for .* from <HOST>(\. .* in .*?/rcube_imap\.php on line \d+ \(\S+ \S+\))?$
|
`- Number of matches:
   [1] 14310 match(es)

Ignoreregex
|- Regular expressions:
|
`- Number of matches:

Summary
=======

Addresses found:
[1]
    61.170.8.8 (Thu Dec 06 13:10:03 2012)
    ...[14309 more results in our logs!]

If you see hits, great, that means our regex worked, and you have some failed logins in the logs.
If you don’t get any results, check your log (use grep) and see if the log warning has changed. The regex I’ve posted works for roundcube 0.84

Once you’re happy, edit jail.conf, enable the plugin.
(set enabled = true), and restart fail2ban with

service fail2ban restart

Looks like Ubuntu 13 has changed the dev id’s for disks!
If you use ZFS, like us, then you may be caught by this subtle naughty change.

Previously, disk-id’s were something like this:
scsi-SATA_ST4000DM000-1CD_Z3000WGF

In Ubuntu 13 this changed:
ata-ST4000DM000-1CD168_Z3000WGF

According to the FAQ in ZFS on Linux, this *isn’t* supposed to change.

http://zfsonlinux.org/faq.html#WhatDevNamesShouldIUseWhenCreatingMyPool

/dev/disk/by-id/: Best for small pools (less than 10 disks)
Summary: This directory contains disk identifiers with more human readable names. The disk identifier usually consists of the interface type, vendor name, model number, device serial number, and partition number. This approach is more user friendly because it simplifies identifying a specific disk.
Benefits: Nice for small systems with a single disk controller. Because the names are persistent and guaranteed not to change, it doesn't matter how the disks are attached to the system. You can take them all out, randomly mixed them up on the desk, put them back anywhere in the system and your pool will still be automatically imported correctly.

So… on a reboot after upgrading a clients NAS, all the data was missing, with the nefarious pool error.
See below:


root@hpnas:# zpool status
pool: nas
state: UNAVAIL
status: One or more devices could not be used because the label is missing
or invalid. There are insufficient replicas for the pool to continue
functioning.
action: Destroy and re-create the pool from
a backup source.
see: http://zfsonlinux.org/msg/ZFS-8000-5E
scan: none requested
config:

NAME STATE READ WRITE CKSUM
nas UNAVAIL 0 0 0 insufficient replicas
raidz1-0 UNAVAIL 0 0 0 insufficient replicas
scsi-SATA_ST4000DM000-1CD_Z3000WGF UNAVAIL 0 0 0
scsi-SATA_ST4000DX000-1CL_Z1Z036ST UNAVAIL 0 0 0
scsi-SATA_ST4000DX000-1CL_Z1Z04QDM UNAVAIL 0 0 0
scsi-SATA_ST4000DX000-1CL_Z1Z05B9Y UNAVAIL 0 0 0


Don’t worry, the data’s still there. Ubuntu has just changed the disk names, so ZFS assumes the disks are broken.

Simple way to fix it is to export the pool, then reimport with the new names.
Our pool is named “nas” in the example below:

root@hpnas:# zpool export nas
root@hpnas:# zpool import -d /dev/disk/by-id nas -f


As you can see, our pool is now a happy chappy, and our data should be back


root@hpnas:# zfs list
NAME USED AVAIL REFER MOUNTPOINT
nas 5.25T 5.12T 209K /nas
nas/storage 5.25T 5.12T 5.25T /nas/storage
root@hpnas:/dev/disk/by-id# zfs list
NAME USED AVAIL REFER MOUNTPOINT
nas 5.25T 5.12T 209K /nas
nas/storage 5.25T 5.12T 5.25T /nas/storage
root@hpnas:/dev/disk/by-id# zpool status
pool: nas
state: ONLINE
scan: none requested
config:

errors: No known data errors


Bit naughty of Ubuntu to do that imho…

Our underlying hardware uses Dell equipment for the most part inside China.
We use Debian as an OS, and Dell has some software available on their linux repo’s specifically tailored for their (often rebranded from other peoples) hardware.
Eg RAC (Remote Access) bits and pieces, RAID hardware, and BIOS updates.

So, enough about why, how do we use their repo?

First add it as a source
echo 'deb http://linux.dell.com/repo/community/deb/latest /' > /etc/apt/sources.list.d/linux.dell.com.sources.list

…then add gpg keys –

gpg --keyserver pool.sks-keyservers.net --recv-key 1285491434D8786F
gpg -a --export 1285491434D8786F | apt-key add -

apt-get update, to make sure that you have the latest repo bits added, then you can install their goodies.

apt-cache search dell will show you whats in their repo. Pick wisely!

Behind the scenes we use ZFS as storage for our offsite backups.
We have backups in 2 separate physical locations + original data on the server(s), as data is mui importante!
ZFS is a rather nice storage file system that improves radically on older RAID based solutions, offering a lot more funky options, like snapshot’s (where the OS can store have multiple versions of files, similar to Time Machine backups), and more importantly compression.

At some point we’ll be deploying a SAN (storage area network) on a blade server in the data center for data using ZFS, using lots of three and four letter acronyms –
ESXi for base OS, then a VM running providing ZFS storage and iSCSI targets with hardware passthru for other VM’s, then other blades in the server doing clustering.
Right now we’re waiting on an LSI SAS card (see more anacronyms!), so we can deploy…, but I digress.

Back to ZFS.

The ZFS version we use now allows flags, yay!, and that means we can choose alternate compression methods.
There is a reasonably newish compression algorithm called LZ4 that is now supported, and it improves both read and write speeds over normal uncompressed ZFS, and some benefits over compressed ZFS using the standard compression algorithm(s).

To quote: “LZ4 is a new high-speed BSD-licensed compression algorithm written by Yann Collet that delivers very high compression and decompression performance compared to lzjb (>50% faster on compression, >80% faster on decompression and around 3x faster on compression of incompressible data)”

First up check if your zfs version supports it:


zpool upgrade -v
This system supports ZFS pool feature flags.

FEAT DESCRIPTION
-------------------------------------------------------------
async_destroy (read-only compatible)
Destroy filesystems asynchronously.
empty_bpobj (read-only compatible)
Snapshots use less space.
lz4_compress
LZ4 compression algorithm support.


Mine does (well duh!)

So…, I can turn on support.

As a note, lz4 is not backward compatible, so you will need to use a ZFS version that supports flags *and* lz4.
At the time of writing nas4free doesn’t support it, zfsonlinux does though, as does omnios, illumos and other solaris based OS’s.

If you aren’t sure, check first with the command above and see if there is support.
Next step is to turn on that feature.

My storage pools are typically called nas or tank
To enable lz4 compression, its a 2 step process.

zpool set feature@lz4_compress=enabled
zpool set compression=lz4

zpool set feature@lz4_compress=enabled nas
zpool set compression=lz4 nas
zpool set compression=lz4 nas/storage


Once the flag is set though, you can set compression on the pools or volume. If you set at the storage volume level, then new pools inherit the compression setting.

Here are my volumes / pools

zfs list
NAME USED AVAIL REFER MOUNTPOINT
nas 6.09T 4.28T 209K /nas
nas/storage 6.09T 4.28T 6.09T /nas/storage

I’ve already set compression on (although it doesn’t take effect till I copy new data onto the pools / volumes).
We can check compression status by doing a zfs get all command, and filtering by compress

zfs get all | grep compress
nas compressratio 1.00x -
nas compression lz4 local
nas refcompressratio 1.00x -
nas/storage compressratio 1.00x -
nas/storage compression lz4 local
nas/storage refcompressratio 1.00x -


If I create a new pool you’ll see it gets created with the same compression inherited from its parent storage volume.

zfs create nas/test
root@nas:/nas# zfs get all | grep compress
nas compressratio 1.00x -
nas compression lz4 local
nas refcompressratio 1.00x -
nas/storage compressratio 1.00x -
nas/storage compression lz4 local
nas/storage refcompressratio 1.00x -
nas/test compressratio 1.00x -
nas/test compression lz4 inherited from nas
nas/test refcompressratio 1.00x -

I’ll copy some dummy data onto there, then recheck.

nas/test compressratio 1.71x -
nas/test compression lz4 inherited from nas
nas/test refcompressratio 1.71x -

Nice!

Obviously, compression ratio’s will depend highly on the data, but for our purposes, most things are web data, mail and other things, so we’re heavy on text content, and benefit highly from compression.

Once we get our SAN up and running, I’ll be looking at whether I should be using rsync still or I should look at zfs snapshots -> zfs storage on other servers.
That though, is a topic for another day.