Archive for the ‘Linux’ Category

Python zipfile speedup tips

Saturday, October 15th, 2011

I have been working on a django project that requires large zip files to be unzipped.

At first I was just Popen’ing unzip. but its hard to track the progress of extraction, in the case of large files.

So I decided to use pythons zipfile module, and override extractall with a progress callback.

However I was very disappointed with the performance. A major slowdown compared to unzip binary.

Here was unzip performance:
time unzip -q /mnt/files/test.zip

real 0m8.880s
user 0m1.560s
sys 0m0.570s

8 seconds, not bad

This was my test script:

from zipfile import ZipFile
zf = ZipFile("/mnt/files/test.zip")
zf.extractall()

time python test.py

real 6m50.938s
user 0m2.990s
sys 0m1.010s

7 minutes!! what is going on… I scratched my head.. trying different things..
So I tried an strace.. And it was all clear.

If you pass a filename to ZipFile.. it doesnt open the file in the constructor.. oh no.

It actually saves the filename and on each extract operation, it opens the file, then closes.. for each file in the archive.

Now, on a local filesystem, this isn’t a big problem. However with a remote cifs filesystem opening a file is a lot more expensive, hence the slowdown.

So, an easy optimisation is to open the file and pass ZipFile a file descriptor.

from zipfile import ZipFile
zf = ZipFile(open("/mnt/files/test.zip","r"))
zf.extractall()

time python test.py

real 0m10.071s
user 0m2.550s
sys 0m0.690s

Bingo, just ~10% slower than unzip.

If you are using python 2.6, and easy optimisation is to use unzip.py from python 2.7, it has many optimisations with regard to large files in the archive.

Citrix Xenoscope

Tuesday, April 19th, 2011

Hi All

Just a quick note that if you’re at Citrix synergy, you’ll get a chance to get a preview of a piece of software me and my team have been developing here in Dublin.

Basically it’s a tool that should help troubleshoot problems with XenServer installations.

Ian will be giving a workshop in Synergy San Francisco.

It’s a really fun and useful product and I hope you like it!

Logo...

Easy NAT

Tuesday, December 15th, 2009

This is the easy way to give virtual machines access to an external network using iptables.

Warning this is very open and probably shouldnt be used in a production environment.

echo 1 > /proc/sys/net/ipv4/ip_forward

iptables -t nat -A POSTROUTING -j MASQUERADE

Testing QTM

Wednesday, December 2nd, 2009

Just testing QTM, Now using charm

Python winbind/libwbclient module

Friday, November 13th, 2009

I’ve created a python ctypes-based module for interfacing with samba libwbclient, using ctypes means that it doesn’t require compilation, but this does rely on the ABI be compatible between versions of libwbclient, if you notice strange behavior or random segfaults, contact me.

Its tested on Linux but should in theory work on solaris, or any OS with python ctypes and samba 3.2.

Overview of the class:

class WBClient(__builtin__.object)

Class that interfaces with libwbclient via ctypes.

Optionally pass in a domain name, if no domain name is supplied the

default domain is used(from libwbclient).

Methods defined here:

__init__(self, default_domain=None)

authenticate_user(self, username, password, domain=None)

Authenticates a user

If the winbind seperator is in the username False is returned

This means username\password doesnt work, one must explicitly give

the domain as the 3rd argument if not authenticating off the default domain

Returns the full username in the form of Domain\Username (or a different winbind separator) if successful, None if not.

change_user_password(self, username, oldpw, newpw)

Change user password, (Working with samba 3.4)

get_display_name(self, username, domain=None)

Gets the display name of a user/group

get_group_membership(self, username, domain=None)

This function looks up the group membership of a user,

Returns a list if successful, None if not

get_sid(self, name, domain=None)

Get the SID string of a user/group/machine

list_groups(self, domain=None)

Lists the groups in a domain

list_users(self, domain=None)

Lists the users in a domain

lookup_domain_controller(self, domain=None)

Lookup domain controller for a domain(Could not get to work)

reload_interface(self)

(re)loads Interface details, winbind separator etc

resolve_wins_by_ip(self, ip)

Resolve an IP to a Netbios name

resolve_wins_by_name(self, name)

Resolve a Netbios name to an IP.

Get your copy from http://dmarkey.com/svn/winbind

Poor Xorg performance on OpenSuse 11.1

Thursday, January 8th, 2009

I upgraded my opensuse 11 installation on my laptop (Dell XPS 1330) to opensuse 11.1 and I notices straight away that my 2d graphics performance was _abysmal_.

The gfx card on my laptop is Intel Corporation Mobile GM965/GL960 using the xorg intel driver.

Things like scrolling in gmail and scrolling man pages in konsole and 2d screen savers all went to a crawl and the driver kept on crashing the machine, I tried to tweak different things in xorg.conf but nothing helped.

I put up with it for a few weeks but today I got fed up and reverted to 11.0, now the performance is back to acceptible levels.

Because of this i’ve been disappointed with the 11.1 release. Luckily from a Kde 3.5 point of view there is little difference between 11.0 ans 11.1.

Hopefully this issue will be solved and a patch released. Actually come to think of it I really should raise a bug.

If anyone else has experienced issues with the intel driver on OpenSuse 11.1 or xorg 7.4 please leave a comment describing your problems.

Ubuntu Intrepid Xen DomU installer

Thursday, December 4th, 2008

I’ve hacked the ubuntu intrepid(amd64 only) initrd so you can run the installer in a Xen DomU. No more debootstrap!

The only glitch is that the installer thinks that the xen block devices are part of a SATA array, when asked to activate the array select no.

vmlinuz

initrd

Example config:

memory = 500
name = “intrepid”
vcpus = 1
disk = ['phy:/dev/xenvg/ubuntu-test,xvda,w']
kernel = “/xen/vmlinuz”
ramdisk = “/xen/initrd”
#bootloader=”/usr/bin/pygrub”
vif  = [ 'bridge=eth0']
#this is for your framebuffer if needed
#vfb         = ['type=vnc,vncunused=1,vncdisplay=0,vnclisten=0.0.0.0,vncpasswd=password']

When the install is finished uncomment pygrub and comment out kernel and ramdisk and it should be fully functional. If you have any issues or comments please feel free.

Ubuntu intrepid Xen DomU made easy

Thursday, November 6th, 2008

I made this script for bootstrapping and “Xenifying” an ubuntu easily. You will have a working ubuntu domU in 30 seconds flat with a fast mirror.

Root password will always be set to ‘password’. Hostname will be ubuntu , loopback interface will be set up. The sole argument is either a filename or a block device to install the distribution onto. See the top of the script for more configurable options. Notibly the mirror option.

The only prerequisite is that you have (a recent)debootstrap installed. Its available for nearly every distro these days.

Remember that i think you will need Xen 3.3 to boot an ubuntu intrepid DomU.

mkubuntu

How to create an ubuntu server DomU using the normal ubuntu install procedure

Wednesday, September 10th, 2008

How to create an ubuntu server DomU using the normal ubuntu install procedure(64 bit only, 32 bit maybe soon.). It may work or it may not, no guarantees. It did for me.

First download the kernel and the initrd and the ubuntu-8.04.1-server-amd64.iso file from ubuntu.com .

Create the xen config file similar to this one paying attention to all the paths.

Xen Config File:

memory = 256
name = “ubuntutemp”
disk = ['phy:/dev/filevg/ubuntutemp,hda,w','file:/mnt/filelv/Downloads/ISOs/ubuntu-8.04.1-server-amd64.iso,sda:cdrom,r']
kernel = ‘/xen/ubuntutemp/kernel-xen’
ramdisk = ‘/xen/ubuntutemp/initrd-xen’
#bootloader=”/usr/bin/pygrub”
vif  = [ '' ]

Fire up the DomU

xm create -c ubuntu.cfg

Setting up the locale settings will be 1st thing.

Then you will be asked:

Load CD-ROM drivers from a driver floppy? say no
Manually select a CD-ROM module and device? say yes
Select “none”
enter /dev/sda
continue without kernel modules.
Ignore “Could not get identity of device” errors, you will get a few, otherwise continue with the setup normally.

Make sure you have a valid internet connection configured because this will be needed later on in the process

Continue the setup until you get the installation complete dialog that says this:

┌│                         Installation complete                         │
││ Installation is complete, so it is time to boot into your new system. │
││ Make sure to remove the installation media (CD-ROM, floppies), so     │
││ that you boot into the new system rather than restarting the          │
││ installation.                                                         │
││                                                                       │
└│     <Go Back>                                          <Continue>     │


DO NOT
select continue. Select go back.

and from the next dialog select “execute a shell” and select continue.

Then issue the following commands to “xenify” the setup.

chroot /target /bin/bash
aptitude install -y linux-image-xen ##this is where you need your internet connection
sed -ibak ’s/generic/xen/g’ /boot/grub/menu.lst
sed -ibak ’s/splash/console=xvc0/g’ /boot/grub/menu.lst
mv /etc/event.d/tty1 /etc/event.d/xvc0
rm /etc/event.d/tty*
sed -i ’s/tty1/xvc0/g’ /etc/event.d/xvc0
sync
exit
poweroff

Probably best to do these 1 by 1, its important every single command finishes.

Now we need to edit the configuration file to this:

memory = 256
name = “ubuntutemp”
disk = ['phy:/dev/filevg/ubuntutemp,hda,w','file:/mnt/filelv/Downloads/ISOs/ubuntu-8.04.1-server-amd64.iso,sda:cdrom,r']
#kernel = ‘/xen/ubuntutemp/kernel-xen’
#ramdisk = ‘/xen/ubuntutemp/initrd-xen’
bootloader=”/usr/bin/pygrub”
vif  = [ '' ]

Obviously changing paths as needed.

And then fire up the machine again

xm create -c ubuntu.cfg

After a short while you should be presented with a lovely login dialog:

Ubuntu 8.04.1 ubuntu xvc0

ubuntu login:

And thats it, you have a fully functioning ubuntu DomU using the starndard ubuntu server setup procedure. FYI LVM setups have been tested and worked too, Encrypted LVM hasnt been tested.

Installing an ubuntu DomU using the native installer

Wednesday, September 10th, 2008

So I wanted to install an ubuntu DomU in my Centos5 Dom0 which proved to be quite the task due to the lack of debootstrap. So an idea was to get the ubuntu install initrd and modify it so it would work with Xen so the installer would work like it would in a normal system. I succeded to an extent, with a few minor operations at the end of the install its reletively simple to install ubuntu using this system.

I’ll post my intructions and kernel/initrd later.


Copyright © 2018 All Rights Reserved.
No computers were harmed in the 0.367 seconds it took to produce this page.

dmarkey.com