Archive for category python
Post compile hooks with rebar
Posted by Hyperthunk in builds, erlang, python on July 13, 2010
Sometimes you want your build to do some additional work after compiling. For example, in my Erlang implementation of hamcrest, I dynamically generate the main header file after compiling to ensure that all the functions exported from the hamcrest_matchers module are imported when the header gets included and to ensure that wrapper macros for eunit are also generated for each of them.
Typically you’d just have a makefile deal with this, maybe like so:
compile:
./rebar compile; escript header_gen
But that kind of breaks other people’s builds if they want to use rebar themselves. The problem is that rebar doesn’t yet provide a mechanism for overriding the build/compile command for dependencies (at all, let alone individually) and thus if your build requires a makefile to be run then your consumers build has to incur the same kind of complexity. Fortunately there is a work-around. Rebar provides a compile_post_script hook which you can specify in your rebar.config file. You specify a shell command and rebar will run this hook immediately after compilation. Unfortunately it also runs it when the compile command is applied to all your dependencies (again this is something which isn’t controllable yet) which typically will break because your shell command is not relevant to these apps. I spent a little time trying to do something clever with erl --eval '...' but in the end that didn’t play nicely at all. I then tried to encode the whole thing in a compound shell command like if [ -f hrlgen ]; then escript hrl; fi but then discovered that rebar_utils:sh/1 (which runs the shell command) wraps the supplied command text in an exec system call (which obviously isn’t going to work for a complex if..then expression). In the end, it was a comedic reptile that came to the rescue:
{compile_post_script,
"env ERL_LIBS=deps python -c \"import os; os.path.isfile('hrlgen') and os.system('escript hrlgen') or None\""}.
testing django with badrabbit
Posted by Hyperthunk in badrabbit, open source, python, TDD on April 4, 2009
One of the design decisions I made when writing badrabbit was to make it fully compatible with unittest (one of the two unit testing frameworks that come with the python stdlib). My reasoning was twofold, firstly to minimize the amount of disruption caused by having to learn a new API and secondly, to give folks a chance to integrate badrabbit‘s mocking library and test decorator syntax in to existing tests with a minimum of fuss.
This decision paid off recently when I tried integrating badrabbit unit tests into a django application. Django has built in support for running unit tests as part of the development lifecycle. The manage.py script that gets generated when you run django’s equivalent of Rails’ scaffolding, takes one of a several commands and one of these is ./manage.py test <appname>.
This has the effect of going off a running all your tests, where the tests for an application are either in your model definition (models.py) a file named tests.py, or contained in (multiple) modules in a package of the same name. By default as described here, django will go off and look for tests defined with the unittest and doctest frameworks, both of which are part of python‘s stdlib. You can override this behavior by providing your own test runner and doing so is, in fact, ridiculously easy. Nevertheless, this isn’t neccessary because badrabbit is 100% compatible with unittest. And so all you need do to use it in your tests is put and import statement in your script, subclass badrabbit.testmagic.autotest instead of unittest.TestCase and you’re off. Here’s an abridged example:
#!/usr/bin/env python
import badrabbit
from badrabbit import *
from users.models import ServiceUser
class ServiceUserTests(autotest):
@test
def it_should_set_absolute_uri_correctly(self):
user = ServiceUser(username='Johannes')
assert_that(user.get_absolute_url(),
equal_to("/users/Johannes/"))
Of course, django has its own subclass of unittest.TestCase already, making subclassing badrabbit.testmagic.autotest less attractive (not least since they share the same superclass!) so there is another way. If you were to go take a peek at the implementation of badrabbit‘s autotest class you would soon notice that it does nothing apart from setting the metaclass to AutoTest:
class AutoTestCase(unittest.TestCase):
__metaclass__ = AutoTest
autotest = AutoTestCase
So instead of subclassing autotest you can just as well set the metaclass (in exactly this way) on your own test class which can then happily subclass django’s TestCase instead.
I can’t imagine the integration being much simpler really. Because badrabbit depends on hamcrest-python, which doesn’t appear to come with an installer, we stuck it a couple of directories above our code and run the manage.py utility using a wrapper shell script which configures the PYTHON_PATH environment variable to include this first.
moving on with python unit testing
Posted by Hyperthunk in badrabbit, python, pyunit on January 27, 2009
My earlier attempts to put some tasty wrap around pyunit have come to fruition. In an early version but hopefully somewhat usable nonetheless, is badrabbit. I hope it proves useful to others as well as to me.
Python Homework
Posted by Hyperthunk in badrabbit, python, pyunit on April 22, 2008
We generate our Python SDK using our (sort of) open source SDK generation system, which is hosted by our friends at osmosoft. You can find the SDK generation code here.
Anyway, there has been much debate about langauge proliferation within our group. As one of the most enthusiastic exponents of a polyglot programming, I perform quite a bit of evangelism within our group. Most recently, I’ve started the work of setting up a Python focus group, and in the course of doing so, have begun setting some “homework” for some of my keen fellow engineers.
Always ahead of the game, Kerry has already started blogging about some of these exploits. The second exercise is an interesting one. I’ll discuss the thinking behind the exercise, followed by my solution.
Here’s the exercise itself:
Write a recursive function that calculates the value of Fibonacci numbers. These are your acceptance criteria:
- Calculating fib(250) must return 7896325826131730509282738943634332893686268675876375
- The function must use recursion. No intermediary data structures, etc.
- The implementation must be written in pure python – no C extension modules, that’s cheating.
- The function must calculate the 250th Fibonacci number in under one second.
You will get extra points if:
- You can also demonstrate a proof for the Reciprocal Fibonacci constant, meeting the following conditions
- Your proof must also run in under one second
- Your proof must not duplicate any of the concerns addressed by your original Fibonacci function implementation.
- Your proof is allowed to call into the Fibonacci function though!
The reciprocal Fibonacci constant is defined here.
Here’s a couple of hints about things to look for:
- Memoization
- Function decorators
Well now. What’s the point of this exercise?
The point here is to get people thinking about function composition. Re-use is so often quoted as being important in our work, and yet in our industry at large, we often see developers failing to apply even basic functional (de!)composition, let alone OOP and other high level programming paradigms.
Properly composed functions are a basic unit of abstraction in most languages, and Python is no exception.
Implementation
The performance requirements of the spec are really a red herring. The key to this exercise is to avoid duplicating the performance enhancing code, which is required to avoid the slowdown (and possible program failure) resulting from excessive recursion. I decided to abstract a simple caching algorithm as a function decorator, which is then applied to both recursive functions (fibonacci and the fib’ constant implementation). You could use the ‘memoized’ decorator anywhere you have a function that is referentially transparent. Here’s the code:
#!/usr/bin/env python def memoized(fun): target = fun cache = {} def memoized(*args, **kwargs): if cache.has_key(args[0]): return cache[args[0]] result = target(*args, **kwargs) cache[args[0]] = result return result return memoized @memoized def fib(n): if n < 2: return n return fib(n - 1) + fib(n - 2) @memoized def fibConstant(n): if(n == 1): return (1.0 / fib(n)) else: return (1.0 / fib(n)) + fibConstant(n - 1.0) if __name__=='__main__': import profile profile.run('print fib(250)') profile.run('print fibConstant(93.0)')
Pyunit
Posted by Hyperthunk in JUnit, python, pyunit, TDD on April 16, 2007