Archive for category python

Post compile hooks with rebar

Sometimes you want your build to do some additional work after compiling. For example, in my Erlang implementation of hamcrest, I dynamically generate the main header file after compiling to ensure that all the functions exported from the hamcrest_matchers module are imported when the header gets included and to ensure that wrapper macros for eunit are also generated for each of them.

Typically you’d just have a makefile deal with this, maybe like so:

compile:
./rebar compile; escript header_gen

But that kind of breaks other people’s builds if they want to use rebar themselves. The problem is that rebar doesn’t yet provide a mechanism for overriding the build/compile command for dependencies (at all, let alone individually) and thus if your build requires a makefile to be run then your consumers build has to incur the same kind of complexity. Fortunately there is a work-around. Rebar provides a compile_post_script hook which you can specify in your rebar.config file. You specify a shell command and rebar will run this hook immediately after compilation. Unfortunately it also runs it when the compile command is applied to all your dependencies (again this is something which isn’t controllable yet) which typically will break because your shell command is not relevant to these apps. I spent a little time trying to do something clever with erl --eval '...' but in the end that didn’t play nicely at all. I then tried to encode the whole thing in a compound shell command like if [ -f hrlgen ]; then escript hrl; fi but then discovered that rebar_utils:sh/1 (which runs the shell command) wraps the supplied command text in an exec system call (which obviously isn’t going to work for a complex if..then expression). In the end, it was a comedic reptile that came to the rescue:

{compile_post_script,
"env ERL_LIBS=deps python -c \"import os; os.path.isfile('hrlgen') and os.system('escript hrlgen') or None\""}.

2 Comments

testing django with badrabbit

One of the design decisions I made when writing badrabbit was to make it fully compatible with unittest (one of the two unit testing frameworks that come with the python stdlib). My reasoning was twofold, firstly to minimize the amount of disruption caused by having to learn a new API and secondly, to give folks a chance to integrate badrabbit‘s mocking library and test decorator syntax in to existing tests with a minimum of fuss.

This decision paid off recently when I tried integrating badrabbit unit tests into a django application. Django has built in support for running unit tests as part of the development lifecycle. The manage.py script that gets generated when you run django’s equivalent of Rails’ scaffolding, takes one of a several commands and one of these is ./manage.py test <appname>.

This has the effect of going off a running all your tests, where the tests for an application are either in your model definition (models.py) a file named tests.py, or contained in (multiple) modules in a package of the same name. By default as described here, django will go off and look for tests defined with the unittest and doctest frameworks, both of which are part of python‘s stdlib. You can override this behavior by providing your own test runner and doing so is, in fact, ridiculously easy. Nevertheless, this isn’t neccessary because badrabbit is 100% compatible with unittest. And so all you need do to use it in your tests is put and import statement in your script, subclass badrabbit.testmagic.autotest instead of unittest.TestCase and you’re off. Here’s an abridged example:

#!/usr/bin/env python

import badrabbit
from badrabbit import *
from users.models import ServiceUser

class ServiceUserTests(autotest):
    @test
    def it_should_set_absolute_uri_correctly(self):
        user = ServiceUser(username='Johannes')
        assert_that(user.get_absolute_url(),
            equal_to("/users/Johannes/"))

Of course, django has its own subclass of unittest.TestCase already, making subclassing badrabbit.testmagic.autotest less attractive (not least since they share the same superclass!) so there is another way. If you were to go take a peek at the implementation of badrabbit‘s autotest class you would soon notice that it does nothing apart from setting the metaclass to AutoTest:

class AutoTestCase(unittest.TestCase):
    __metaclass__ = AutoTest
autotest = AutoTestCase

So instead of subclassing autotest you can just as well set the metaclass (in exactly this way) on your own test class which can then happily subclass django’s TestCase instead.

I can’t imagine the integration being much simpler really. Because badrabbit depends on hamcrest-python, which doesn’t appear to come with an installer, we stuck it a couple of directories above our code and run the manage.py utility using a wrapper shell script which configures the PYTHON_PATH environment variable to include this first.

Leave a comment

moving on with python unit testing

My earlier attempts to put some tasty wrap around pyunit have come to fruition. In an early version but hopefully somewhat usable nonetheless, is badrabbit. I hope it proves useful to others as well as to me.

Leave a comment

Python Homework

At work, Python is one of many languages we utilize. In fact, Python is one of the APIs we offer for accessing our services.

We generate our Python SDK using our (sort of) open source SDK generation system, which is hosted by our friends at osmosoft. You can find the SDK generation code here.

Anyway, there has been much debate about langauge proliferation within our group. As one of the most enthusiastic exponents of a polyglot programming, I perform quite a bit of evangelism within our group. Most recently, I’ve started the work of setting up a Python focus group, and in the course of doing so, have begun setting some “homework” for some of my keen fellow engineers.

Always ahead of the game, Kerry has already started blogging about some of these exploits. The second exercise is an interesting one. I’ll discuss the thinking behind the exercise, followed by my solution.

Here’s the exercise itself:

Write a recursive function that calculates the value of Fibonacci numbers. These are your acceptance criteria:

  • Calculating fib(250) must return 7896325826131730509282738943634332893686268675876375
  • The function must use recursion. No intermediary data structures, etc.
  • The implementation must be written in pure python – no C extension modules, that’s cheating.
  • The function must calculate the 250th Fibonacci number in under one second.

    You will get extra points if:

  • You can also demonstrate a proof for the Reciprocal Fibonacci constant, meeting the following conditions
    • Your proof must also run in under one second
    • Your proof must not duplicate any of the concerns addressed by your original Fibonacci function implementation.
    • Your proof is allowed to call into the Fibonacci function though!

The reciprocal Fibonacci constant is defined here.

Here’s a couple of hints about things to look for:

  • Memoization
  • Function decorators

Well now. What’s the point of this exercise?

The point here is to get people thinking about function composition. Re-use is so often quoted as being important in our work, and yet in our industry at large, we often see developers failing to apply even basic functional (de!)composition, let alone OOP and other high level programming paradigms.

Properly composed functions are a basic unit of abstraction in most languages, and Python is no exception.

Implementation

The performance requirements of the spec are really a red herring. The key to this exercise is to avoid duplicating the performance enhancing code, which is required to avoid the slowdown (and possible program failure) resulting from excessive recursion. I decided to abstract a simple caching algorithm as a function decorator, which is then applied to both recursive functions (fibonacci and the fib’ constant implementation). You could use the ‘memoized’ decorator anywhere you have a function that is referentially transparent. Here’s the code:

#!/usr/bin/env python

def memoized(fun):
  target = fun
  cache = {}
  def memoized(*args, **kwargs):
      if cache.has_key(args[0]):
          return cache[args[0]]

      result = target(*args, **kwargs)
      cache[args[0]] = result
      return result
  return memoized

@memoized
def fib(n):
  if n < 2: return n
  return fib(n - 1) + fib(n - 2)

@memoized
def fibConstant(n):
  if(n == 1):
      return (1.0 / fib(n))
  else:
      return (1.0 / fib(n)) + fibConstant(n - 1.0)

if __name__=='__main__':
  import profile
  profile.run('print fib(250)')
  profile.run('print fibConstant(93.0)')

2 Comments

Pyunit

One of the things I like about JUnit 4 is to see anotations replace the naming convention required by earlier versions. I like the declarative approach so much, in fact, that I’ve extended pyunit (the standard python unit testing framework) with some simple hacks that yield the same syntax. By including the pyunit2.py module, you can now write your test methods like this:

Leave a comment

Design a site like this with WordPress.com
Get started