Okey, there is just travis complaining with pypy, can you look at this?

Sorry for the slow reply, I'm a little bit confused by what's going on with pypy but I will fix it - I've just been away for a few days and will look at this tomorrow.

The issue is the different GC algos each use. CPython is keeping the handle alive much longer than pypy is (or pypy is moving the object, but that's less likely here, I think).

When you assign callbacks.payload = ffi.handle(self), the python object immediately leaves the scope of python objects and is free to be reclaimed, which pypy does in this situation, making the payload/data handle an invalid pointer, which looks like it's what pypy tries to tell you.

This is the reason why the code in master assigns self to a pointer of itself, so callbacks.payload does not become unreferenced. Depending on how paranoid we want to be about GC, we have to assign the handle first to a variable, or attach it to self itself (temporarily). Doing

-        callbacks.payload = ffi.new_handle(self)
+        self_handle = ffi.new_handle(self)
+        callbacks.payload = self_handle

is enough for it to pass right now. I'm not sure if there's any rules about python variables being kept alive until the end of the function or not. If not, we might want to do

self._self_handle = ffi.new_handle(self)
callbacks.payload = self._self_handle
... # fetch and stuff
del self._self_handle

to make sure that the handle doesn't become unreferenced on the python side for as long as the callbacks struct is alive.

Oh right, thanks for the explanation! I failed to realise that because callbacks doesn't behave exactly like a Python object, pypy doesn't keep the members alive for me. It's probably right to be a bit paranoid too - even though self_handle would stay in scope, if there's no reference to it then Pypy may be within its rights to clean it up in the absence of a with.

Yeah, it's a similar issue as with a git_strarray. We also need to keep the references around as even CPython will remove them (presumably because it's across a function return).