I have been working on a django project that requires large zip files to be unzipped.

At first I was just Popen’ing unzip. but its hard to track the progress of extraction, in the case of large files.

So I decided to use pythons zipfile module, and override extractall with a progress callback.

However I was very disappointed with the performance. A major slowdown compared to unzip binary.

Here was unzip performance:
time unzip -q /mnt/files/test.zip

real 0m8.880s
user 0m1.560s
sys 0m0.570s

8 seconds, not bad

This was my test script:

from zipfile import ZipFile
zf = ZipFile("/mnt/files/test.zip")
zf.extractall()

time python test.py

real 6m50.938s
user 0m2.990s
sys 0m1.010s

7 minutes!! what is going on… I scratched my head.. trying different things..
So I tried an strace.. And it was all clear.

If you pass a filename to ZipFile.. it doesnt open the file in the constructor.. oh no.

It actually saves the filename and on each extract operation, it opens the file, then closes.. for each file in the archive.

Now, on a local filesystem, this isn’t a big problem. However with a remote cifs filesystem opening a file is a lot more expensive, hence the slowdown.

So, an easy optimisation is to open the file and pass ZipFile a file descriptor.

from zipfile import ZipFile
zf = ZipFile(open("/mnt/files/test.zip","r"))
zf.extractall()

time python test.py

real 0m10.071s
user 0m2.550s
sys 0m0.690s

Bingo, just ~10% slower than unzip.

If you are using python 2.6, and easy optimisation is to use unzip.py from python 2.7, it has many optimisations with regard to large files in the archive.

This checks each file extracted for “..” in the path, and dont go over a file size limit that you set in the constructor.

from zipfile import ZipFile, ZipInfo
import os

class NotSafeFileException(Exception):
    pass


class SafeZipFile(ZipFile):
    def __init__(self, *args, **kwargs):
        self.max_size = kwargs.pop('max_size', None)
        ZipFile.__init__(self, *args, **kwargs)

    def extract(self, member, path=None, pwd=None):
        if not isinstance(member, ZipInfo):
            member = self.getinfo(member)
        if path is None:
            path = os.getcwd()
        self.safety_check(member)
        return self._extract_member(member, path, pwd)

    def safety_check(self, zipinfo):
        """Make sure that the file/dir:
            * Doesn't start with a slash in the path
            * Doesnt have ".." in the path
            * If max_size is passed, make sure the file isnt bigger than that threshhold
        """
        if zipinfo.filename.startswith("/"): raise NotSafeFileException("%s starts with a slash" % tarinfo.path)
        if ".." in zipinfo.filename: raise NotSafeFileException("%s contains '..'" % zipfile.filename)
        if self.max_size and self.max_size < zipinfo.file_size: raise NotSafeFileException("%s is too big" % zipinfo.filename)

This TarFile module is a drop-in replacement for TarFile which makes sure that files in a tarfile, is safe using the following criteria:

* Doesn’t start with a slash in the path
* Doesnt have “..” in the path
* Is either a normal file or directory(no fifos, symlinks)
* If max_size is passed, make sure the file isnt bigger than that threshhold

Tested on 2.7

from tarfile import TarFile, ExtractError
import copy
import operator
import os.path

class NotSafeFileException(Exception):
    pass

class SafeTarFile(TarFile):
    def __init__(self, *args, **kwargs):
        self.max_size = kwargs.pop('max_size', None)
        super(SafeTarFile,self).__init__(*args, **kwargs)

    def safety_check(self, tarinfo, max_size=None):
        """Make sure that the file/dir:
            * Doesn't start with a slash in the path
            * Doesnt have ".." in the path
            * Is either a normal file or directory(no fifos,  symlinks)
            * If max_size is passed, make sure the file isnt bigger than that threshhold
        """

        if tarinfo.path.startswith("/"): raise NotSafeFileException("%s starts with a slash" % tarinfo.path)
        if ".." in tarinfo.path: raise NotSafeFileException("%s contains '..'" % tarinfo.path)
        if not tarinfo.isfile() and not tarinfo.isdir(): raise NotSafeFileException("%s is a strange filetype" % tarinfo.name)
        if self.max_size and self.max_size < tarinfo.size: raise NotSafeFileException("%s is too big" % tarinfo.name)

    def extract(self, member, path=""):
        self._check("r")

        if isinstance(member, basestring):
            tarinfo = self.getmember(member)
        else:
            tarinfo = member
        self.safety_check(tarinfo)
        super(SafeTarFile, self).extract(tarinfo, path)

This is a quick Monkeypatch to make sure that all child processes are killed when a worker is killed.

from celery.worker import state
import celery.worker.job
from celery import exceptions

def kill_child_processes(parent_pid, sig=signal.SIGTERM, top=True):
        ps_command = subprocess.Popen("ps -o pid --ppid %d --noheaders" % parent_pid, shell=True, stdout=subprocess.PIPE)
        ps_output = ps_command.stdout.read()
        retcode = ps_command.wait()
        if retcode != 0: return
        for pid_str in ps_output.split("n")[:-1]:
            try:
                kill_child_processes(int(pid_str), sig, top=False)

                if not top: os.kill(int(pid_str), sig)
            except OSError: pass


def on_timeout(self, soft, timeout):
    """Handler called if the task times out."""
    state.task_ready(self)
    if soft:
        self.logger.warning("Soft time limit (%ss) exceeded for %s[%s]" % (
            timeout, self.task_name, self.task_id))
        exc = exceptions.SoftTimeLimitExceeded(timeout)
    else:
        kill_child_processes(self.worker_pid)
        self.logger.error("Hard time limit (%ss) exceeded for %s[%s]" % (
            timeout, self.task_name, self.task_id))
        exc = exceptions.TimeLimitExceeded(timeout)

    if self._store_errors:
        self.task.backend.mark_as_failure(self.task_id, exc)


celery.worker.job.TaskRequest.on_timeout = on_timeout

Hi All

Just a quick note that if you’re at Citrix synergy, you’ll get a chance to get a preview of a piece of software me and my team have been developing here in Dublin.

Basically it’s a tool that should help troubleshoot problems with XenServer installations.

Ian will be giving a workshop in Synergy San Francisco.

It’s a really fun and useful product and I hope you like it!

So in work(Citrix),

We’ve been working on a django based application to analyse debug data from XenServer..

I was asked to deploy the application internally in such a way that availed of AD Single sign on in our intranet..  This would mean that I’d have to run it on a windows box and avail of Kerberos or NTLM authentication..

This posed a problem for me.. My application depends a lot on some standard Linux commands(grep, busybox etc)

So I decided to keep the application running on a Linux machine, and try to use some protocol to proxy the requests from IIS, which would be doing the authentication.. I would use the REMOTE_USER header to identify the user authenticated.

I tried apache with fastcgi and mod_auth_sspi.. but the authentication was flaky, sometimes worked, sometimes didn’t..

I decided that I needed to use IIS if I was to get stable authentication… But its fastcgi implementation didnt support sending the request to another box.

I was running out of ideas.. then I seen that there was a AJP connector for IIS which comes from the apache tomcat project in the form of isapi_redirect. So I gave it a go..

I used flups ajp implementation on the django side.. everything was going great, I could authenticate using IIS and the request was passwd back to the django box..

Except for one thing.. My application requires a large file upload(~100mb).. and it took hours.. i.e. 20kb/s. I first thought it was IIS.. but no. Flups AJP implementation is very very slow.

I seen that flups author also had a C based project, ajp-wsgi, I tried it out and it was much better very reliable.. there was a small hick-up at the start but allen patched that up.. Thanks..

So in essence, If you need windows authentication, and you need your django app to run on Linux.. use IIS, isapi_redirect, and ajp-wsgi..

The IPhone is a great device…

Great operating system…

Readily hackable…

Great app store…

But.. IT DOESN’T HAVE AN FM RADIO

I tried getting my radio via 3G.. but I always forgot to turn it off and my credit would be all gone.

So I decided to get rid of it.. and I got myself a ZTE Blade(Orange san francisco). It’s such a luxury having an FM radio now.. I don’t think i’ll move off android..

A message for Steve Jobs: Dont be stupid, i’m not going to buy a 400 euro device and it not have an FM radio.. it will cost you like 5 cent to put an FM radio in the device..

The ZTE blade cost me 99 pounds sterling.. it’s not as good in the IPhone in some respects.. but it has the most important feature I need…

Because I’ve been on a OpenWRT buzz, I wanted to use an old D-Link DSL/Router for OpenWRT.

I was ripping my hair out trying to figure out why I couldnt get DSL to sync. It turns out that I was using a variable voltage adaptor and I should have been giving 12V but i was only giving 6V! Anyway once i upped the voltage DSL started to sync.

For anyone else that wants to use OpenWRT with eircom use the following settings in /etc/config/network:

config 'interface' 'loopback'
	option 'ifname' 'lo'
	option 'proto' 'static'
	option 'ipaddr' '127.0.0.1'
	option 'netmask' '255.0.0.0'

config 'interface' 'lan'
	option 'type' 'bridge'
	option 'ifname' 'eth0'
	option 'proto' 'static'
	option 'ipaddr' '192.168.2.1'
	option 'netmask' '255.255.255.0'
	option 'nat' '1'

config 'atm-bridge'
	option 'unit' '0'
	option 'encaps' 'llc'
	option 'vpi' '8'
	option 'vci' '35'
#	option 'payload' 'routed'

config 'interface' 'wan'
	option 'ifname' 'nas0'
	option 'proto' 'pppoe'
	option 'encaps' 'llc'
	option 'vpi' '8'
	option 'vci' '35'
	option 'keepalive' '5,20'
	option 'username' 'eircom@eircom.net'
	option 'password' 'broadband1'

New Xen install initrds.

Over the past week i’ve been working on Ubuntu 9.10 and Arch linux install initrds for Xen PVM.

The arch one has been straight forward but the ubuntu one is a bit on the tricky side(grub2 mainly).

Will post them up when i have them tested a bit.

This is the easy way to give virtual machines access to an external network using iptables.

Warning this is very open and probably shouldnt be used in a production environment.

echo 1 > /proc/sys/net/ipv4/ip_forward

iptables -t nat -A POSTROUTING -j MASQUERADE