aTuple = (1, 2, 3, 4, 5)
print (aTuple[1:3])
print (aTuple[0:7])
print(aTuple[3:1:-1])

which produces:

(2, 3)
(1, 2, 3, 4, 5)
(4, 3)

It doesn’t matter that, in the second instance, the end index is beyond the last index of the string. Also, the end index is excluded.

If you want to go to the start of the tuple, then you need to set the end index to -1:

print(aTuple[3:-1:-1])

But this doesn’t produce the expected result.

( )

This is because negative indexes are from the end of the tuple. So -1 is the last element in the tuple.

print(aTuple[2:-1])

produces:

(3, 4)

Again, this excludes the final index. To include either the start or the end of the tuple as the final element of the result, the end index should be left empty:

print(aTuple[2:])
print(aTuple[2::-1])

giving:

(3, 4, 5)
(3, 2, 1)

Finally, in most cases where you can use a tuple, you can use a list instead:

aList = list(aTuple)

print (aList[1:3])
print (aList[0:7])
print(aList[3:1:-1])
print(aList[3:-1:-1])
print(aList[2:-1])
print(aList[2:])
print(aList[2::-1])

producing similar results as before, but lists instead of tuples:

[2, 3]
[1, 2, 3, 4, 5]
[4, 3]
[]
[3, 4]
[3, 4, 5]
[3, 2, 1]

But this doesn’t work for the % operator:

print("aTuple[0]=%s, aTuple[1]=%s, aTuple[2]=%s, aTuple[3]=%s, aTuple[4]=%s" % aTuple)
print("aList[0]=%s, aList[1]=%s, aList[2]=%s, aList[3]=%s, aList[4]=%s" % aList)

aTuple[0]=1, aTuple[1]=2, aTuple[2]=3, aTuple[3]=4, aTuple[4]=5
Traceback (most recent call last):
  File "c:/prr/cgibin/data/prr/codebright/tuple.py", line 29, in <module>
    print("aList[0]=%s, aList[1]=%s, aList[2]=%s, aList[3]=%s, aList[4]=%s" % aList)
TypeError: not enough arguments for format string

First main_module.py:

g_main_value = None

def get_main_value():
    return g_main_value


def main_test():
    print("sub_value = %s" % get_sub_value())


from sub_module import get_sub_value, sub_test, sub_init

def main_init():
    global g_main_value

    g_main_value = 23


if __name__ == "__main__":
    main_init()
    sub_init()

    main_test()
    sub_test()

    print("main_value (in main_module) = %s" % get_main_value() )

Then sub_module.py:

g_sub_value = None

def get_sub_value():
    return g_sub_value


def sub_test():
    print("main_value (in sub_module) = %s" % get_main_value())


def sub_init():
    global g_sub_value

    g_sub_value = 31


from main_module import get_main_value

The output is:

sub_value = 31
main_value (in sub_module) = None
main_value (in main_module) = 23

What’s happening? A global with 2 different values? Add the following code to the bottom of main_module:

def show_globals(module_name):
    mod = __import__(module_name)
    print(module_name)
    for k in dir(mod):
        if k.startswith("g_"):
            print("  %s: %s" % (k, getattr(mod, k)))
    print("")


if __name__ == "__main__":
    print("")
    show_globals("__main__")
    show_globals("main_module")
    show_globals("sub_module")

The output is:

__main__
  g_main_value: 23

main_module
  g_main_value: None

sub_module
  g_sub_value: 31

This gives us a clue as to what’s happening: the __main__ module may be loaded twice if used by another module. If you wanted to set a global in the __main__ module that is usable by other modules, then one way to do it is:

Add the following to main_module.py

mod = __import__("main_module")
setattr(mod, "g_main_value2", 34)

if __name__ == "__main__":
    print("")
    show_globals("__main__")
    show_globals("main_module")
    show_globals("sub_module")

And the output will be

__main__
  g_main_value: 23
  g_main_value2: None

main_module
  g_main_value: None
  g_main_value2: 34

sub_module
  g_sub_value: 31

<?php

$res = explode(",", "");
print_r($res);

?>

produces:

Array
(
    [0] =>
)

I was expecting (needing?) an empty array. After thinking about it a bit more, it is clear that returning a single empty string is a logical return value. However, different languages have different returns for this special case. Python (2.6.4 and 3.1.2) are both like PHP (5.3.5), but Perl (5.10.1) behaves differently:

For PHP:

<?php
function test_explode($txt)
{
    print("Exploding '$txt' to give [" );
    $sep = "";
    $res = explode(",", $txt);
    foreach ($res as $elt)
    {
        print "$sep'$elt'";
        $sep = ", ";
    }
    print "]\n";

}

test_explode("a,b");
test_explode(",");
test_explode("");

?>

produces:

Exploding 'a,b' to give ['a', 'b']
Exploding ',' to give ['', '']
Exploding '' to give ['']

For Python:

def test_split(txt):
    print("Splitting '%s' to give %s" % (txt, txt.split(",")))

test_split("a,b")
test_split(",")
test_split("")

produces:

Splitting 'a,b' to give ['a', 'b']
Splitting ',' to give ['', '']
Splitting '' to give ['']

For Perl:

sub test_split
{
    my $txt = shift;
    my @res = split(",", $txt);
    print("Splitting '$txt' to give [");
    my $sep = "";
    foreach my $elt (@res)
    {
        print "$sep'$elt'";
        $sep = ", ";
    }
    print "]\n";
}


test_split("a,b");
test_split(",");
test_split("");

produces:

Splitting 'a,b' to give ['a', 'b']
Splitting ',' to give []
Splitting '' to give []

My test data consists of 950,000 lines in 5 gzipped Apache log files. I compared the times using the Python gzip module, and the zcat method suggested in http://bugs.python.org/issue7471.

A quick summary of results: yes, the issue is still relevant. The average times are:

For Python 2.6.4

• gzip.open – 9.5 seconds
• zcat and pipe – 2.3 seconds

For Python 3.1.2

• gzip.open – 11.6 seconds
• zcat and pipe – 2.7 seconds

Below are the details. First, the source code to time the tests:

import os
import sys

if sys.version.startswith("3"):
    import io
    io_method = io.BytesIO
else:
    import cStringIO
    io_method = cStringIO.StringIO

import gzip
import subprocess
import time

dirname = "test"
fl = os.listdir(dirname)

fl.sort()

ttime = [0, 0]
runs = 5

for i in range(2 * runs):
    st = time.time()

    for fn in fl:
        if not fn.endswith(".gz"):
            continue

        cc = 0
        lc = 0
        sz = 0

        fullfn = os.path.join(dirname, fn)
        sz += os.stat(fullfn)[6]
        if i % 2 == 0:
            fh = gzip.GzipFile(fullfn, "r")
        else:
            p = subprocess.Popen(["zcat", fullfn], stdout = subprocess.PIPE)
            fh = io_method(p.communicate()[0])
            assert p.returncode == 0

        for line in fh:
            lc += 1
            cc += len(line)

    et = time.time()
    dt = et - st

    ttime[i % 2] += dt
    print("time-taken = %0.2f seconds, 1000 characters per second = %0.0f, file size per second = %0.0f, character count=%s, line count=%s file size = %s" % (dt, 0.001 * cc / dt, 0.001 * sz / dt, cc, lc, sz))

print("\nAverages")
print("  gzip.open - %0.1f seconds" % (ttime[0] / runs))
print("  zcat and pipe - %0.1f seconds" % (ttime[1] / runs))

Timings for Python 2.6.4:

time-taken = 9.71 seconds, 1000 characters per second = 5237, file size per second = 480, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.28 seconds, 1000 characters per second = 22300, file size per second = 2044, character count=50859394, line count=194151 file size = 4661207
time-taken = 9.48 seconds, 1000 characters per second = 5363, file size per second = 492, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.24 seconds, 1000 characters per second = 22750, file size per second = 2085, character count=50859394, line count=194151 file size = 4661207
time-taken = 9.44 seconds, 1000 characters per second = 5389, file size per second = 494, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.30 seconds, 1000 characters per second = 22156, file size per second = 2031, character count=50859394, line count=194151 file size = 4661207
time-taken = 9.42 seconds, 1000 characters per second = 5397, file size per second = 495, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.26 seconds, 1000 characters per second = 22516, file size per second = 2064, character count=50859394, line count=194151 file size = 4661207
time-taken = 9.61 seconds, 1000 characters per second = 5293, file size per second = 485, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.31 seconds, 1000 characters per second = 22033, file size per second = 2019, character count=50859394, line count=194151 file size = 4661207

Averages
  gzip.open - 9.5 seconds
  zcat and pipe - 2.3 seconds

Timings for Python 3.1.2:

time-taken = 11.65 seconds, 1000 characters per second = 4364, file size per second = 400, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.87 seconds, 1000 characters per second = 17691, file size per second = 1621, character count=50859394, line count=194151 file size = 4661207
time-taken = 11.66 seconds, 1000 characters per second = 4362, file size per second = 400, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.56 seconds, 1000 characters per second = 19834, file size per second = 1818, character count=50859394, line count=194151 file size = 4661207
time-taken = 11.54 seconds, 1000 characters per second = 4408, file size per second = 404, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.64 seconds, 1000 characters per second = 19295, file size per second = 1768, character count=50859394, line count=194151 file size = 4661207
time-taken = 11.58 seconds, 1000 characters per second = 4393, file size per second = 403, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.59 seconds, 1000 characters per second = 19659, file size per second = 1802, character count=50859394, line count=194151 file size = 4661207
time-taken = 11.68 seconds, 1000 characters per second = 4354, file size per second = 399, character count=50859394, line count=194151 file size = 4661207
time-taken = 2.97 seconds, 1000 characters per second = 17123, file size per second = 1569, character count=50859394, line count=194151 file size = 4661207

Averages
  gzip.open - 11.6 seconds
  zcat and pipe - 2.7 seconds

1. enumerate

A built-in function, which allows you to replace code like this:

day_list = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
for day_index in range(len(day_list)):
    day_name = day_list[day_index]
    print(day_index, day_name)

with the more succinct:

day_list = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"]
for day_index, day_name in enumerate(day_list):
    print(day_index, day_name)

2. gzip.GzipFile

I used this whilst processing Apache log files last week. It offers compression to less than 10% of the original uncompressed size, but seems to take the same length of time to read/write, implying a good trade-off between processor use and hard-disk activity. Having changed the open code from

fh = open(<logfile>, "r")

to:

import gzip
...
fh = gzip.GzipFile(<logfile>, "r")

the rest of the code runs unchanged.

Note: if you are running on Linux, then the issue of speed, documented at http://bugs.python.org/issue7471, is relevant, and the zcat approach is faster.
 

3. collections.defaultdict

This provides a dictionary, which initialises a key on first read using a method.

For instance:

import collections
...
zerodict = collections.defaultdict(int)
print(zerodict["key"], zerodict[4])

produces:

0 0

Try and predict the outcome (should work with Python 2.5 – Python 3.1)

import re

abcd = "abcd"
ad = "ad"

cre = re.compile(r"^(a)(bc)?(d)$")

mos1 = cre.match(abcd)
print(cre.sub(r"\1, \2, \3", abcd))

mos2 = cre.match(ad)
print(cre.sub(r"\1, \2, \3", ad))

This produces:

a, bc, d
Traceback (most recent call last): 
  File "<filename>", line <linenumber>, in <module>
    print(cre.sub(r"\1, \2, \3", ad))
  File "C:\programs\Python25\lib\re.py", line 266, in filter
    return sre_parse.expand_template(template, match)
  File "C:\programs\Python25\lib\sre_parse.py", line 793, in expand_template
    raise error, "unmatched group"
sre_constants.error: unmatched group

(Line numbers vary with different versions of Python – this is Python 2.5)

Adding the following code illustrates the problem:

print(mos1.group(1), mos1.group(2), mos1.group(3) )
print(mos2.group(1), mos2.group(2), mos2.group(3) )

For Python 2.5, this outputs

('a', 'bc', 'd')
('a', None, 'd')

So now it is clear why it is failing, what is the solution. First fix was:

cre2 = re.compile(r"^(a)((bc)?)(d)$")
print(cre2.sub(r"\1, \2, \4", ad))

This fix works, but it has side-effects – the last group number has changed. So a better solution might be:

cre3 = re.compile(r"^(a)((?:bc)?)(d)$")
print(cre3.sub(r"\1, \2, \3", ad))

Using the non-capturing group notation, (?:…) the problem is fixed without side-effects.

I’m not entirely happy that the problem occurs in the first place, but it does.

For Ubuntu (Hardy Heron, 8.04), I found xautomation, which worked fine. For many (but not all) of the non alpha-numeric keystrokes, you need to use the name of the key – there is a helpful list here.

I tried xautomation with Fedora 13, but it was not happy. So I had a look around for an alternative. One possibility (I found them hard to find) was xdotool. Of course, the syntax for sending keystrokes is slightly different! It is slightly more consistent in that non-alphanumerics all need to be sent using the name of the key, so the helpful list (above) is still helpful!

I’ll try to post some examples at a later date.

import ctypes

int_array_type = ctypes.c_int * 2
pint_array_type = ctypes.POINTER(ctypes.c_int)


def display_values(pointer_list):
    #print("pointer_list = %s" % pointer_list)
    for i in range(3):
        pint_array = ctypes.cast(pointer_list[i], pint_array_type)

        print("values of pointer_list[%d] are [%d, %d]" % (i, pint_array[0], pint_array[1]) )


def wrong():
    pointer_list = []
    for i in range(3):
        int_array = int_array_type()
        for j in range(2):
            int_array[j] = i + j

        pointer_list.append(ctypes.addressof(int_array))

    display_values(pointer_list)


wrong()

If you’re using Python 2.x or Python 3.x, then you might be surprised. Your values may vary, but I get:

values of pointer_list[0] are [2, 3]
values of pointer_list[1] are [11187064, 0]
values of pointer_list[2] are [2, 3]


(If you’re using IronPython 2.6.1, the code works, and does not display this particular problem).

Uncommenting line 7 gives us a bit more of a clue. Your values will vary, but mine produces:
pointer_list = [11187064, 11187144, 11187064]

It took me ages to work out what was going on here. The problem seems to be that storing the address of some Python variable (in this case int_array) is not enough to keep it from being reused. The addressof method returns an integer, and this is not a Python reference, so the memory can be reused. And ctypes seems to reuse it every other time.

So one way round this problem would be to keep a reference to it in some other way. There are many ways to do this, but one example would be:

import ctypes

int_array_type = ctypes.c_int * 2
pint_array_type = ctypes.POINTER(ctypes.c_int)


def display_values(pointer_list):
    #print("pointer_list = %s" % pointer_list)
    for i in range(3):
        pint_array = ctypes.cast(pointer_list[i], pint_array_type)

        print("values of pointer_list[%d] are [%d, %d]" % (i, pint_array[0], pint_array[1]) )


def right():
    pointer_list = []
    safe_store = []
    for i in range(3):
        int_array = int_array_type()
        for j in range(2):
            int_array[j] = i + j

        pointer_list.append(ctypes.addressof(int_array))
        safe_store.append(int_array)

    display_values(pointer_list)


right()

This produces the correct output:
values of pointer_list[0] are [0, 1]
values of pointer_list[1] are [1, 2]
values of pointer_list[2] are [2, 3]

There are some good articles on it here and here

To my mind, a key requirement of decorators, after the need to be useful, is to be as unintrusive as possible. So the decorated function or method, and more particularly, users of the decorated function or method should not notice the decoration.

Some of the issues, in order of increasing complexity:

• functions and methods
• return values
• function / method arguments and keyword arguments
• attributes such as __name__ and __doc__
• decorator arguments
• multiple decorators
• introspection

I don’t have all the answers yet, and I’ll write more in a later post.

An example implementation shows some of the points of interest. This is a decorator with arguments implemented using a function.

from functools import wraps

def decoratorWithArguments(decoratorArgument = None):
    def wrap(functionOrMethod):
        # Support naive introspection
        @wraps(functionOrMethod)
        def wrappedFunctionOrMethod(*args, **kwargs):
            print("Before %s" % str(decoratorArgument) )
            ret = functionOrMethod(*args, **kwargs)
            print("After %s" % str(decoratorArgument) )
            return ret

        return wrappedFunctionOrMethod
    return wrap


@decoratorWithArguments("Decorator Argument Value")
def functionExample():
    """ functionExample docstring """
    return "functionExample"


class ClassExample:
    @decoratorWithArguments()  # defaults to None
    def methodExample(self):
        """ methodExample docstring """
        return "methodExample"


print(functionExample())
classInstance = ClassExample()
print(classInstance.methodExample())

Notes:

• the wraps function from functools copies across various attributes such as the docstring.
• *args and **kwargs should cope with both positional arguments and keyword arguments
• Because decorators work differently with and without arguments, even though this decorator will work without any arguments, I still need to use () on line 22.

It took me an age to understand how decoratorWithArguments could possibly work. It’s a bit like Russian dolls:

• The outer function, decoratorWithArguments, takes the decorator arguments. It returns the inner function, wrap (a closure).
• Having unwrapped the outer function, we now have the returned inner function (wrap). This can then be called with the function or method that we want to decorate. It returns the inner function, wrappedFunctionOrMethod.
• wrappedFunctionOrMethod can then be called. It will be called in place of the original decorated function or method.

Note that the decoration process takes care of all of the above for us. It’s still nice to understand how it can work.

(See here for more information on Ubuntu releases.)

The download of the packages went fine, but the upgrade froze at about 75%. Which is probably not a good thing, as there is no cancel option at this point. Sure enough, on the reboot, the system was not happy.

It was at this point that I discovered the CD-player on the computer was not working either (a hardware issue rather than anything to do with the upgrade).

Luckily, I had an old copy of 6.06 LTS (Dapper Drake) that I could use to back up important data.

I then looked around for non-CD installation options. There seemed to be 3 options:

• Install from ISO mounted on a partition
• Install from ISO placed on a drive
• Install from the network

This is not an area that I am overly familiar with, so problems that I had are likely to be PEBKAC.

The method that I eventually got working is the install from the network. Here are a few hints:

• The install can appear to hang in places. This usually means that it is taking a long time. I got quicker response times later in the evening
• It offers the option of installing lots of extra software towards the end. Don’t! Because your system will not be left in a working state if anything goes wrong at this point. Just install the necessary software later on, once you have a working system. The install process is much more robust once Ubuntu is installed.
• Take care to get the right linux and initrd.gz files. These exist, for each Ubuntu version, for a reboot from the network, an install from the network, and many other purposes.

I also learned a lot about fdisk and the GRUB (bootloader). GRUB is a very useable piece of software. I will write more about it in a later post.

I managed to get Lucid Lynx installed, but it is too slow on the machine (256MB of memory, so maybe not too surprising). Hardy Heron continues to be my default version, and is pleasantly responsive.