Concept ACK, I like adding the sequence nr as an extra part. Also ok with using 32 bit sequence numbers.

I thought a subscriber can't miss a message.

Edit:

But it can: http://stackoverflow.com/a/15821036. So what can a subscriber do if it detects a missing message?

@promag:
I'm not sure if I would design an listening application that fully rely on getting all notifications. With this PR, you could at least detect if you got all (also check if you got the first) notifications and if not, you could poll/sync your data over RPC.

I thought a subscriber can't miss a message.

The way we are using ZMQ, the sender will never block. This is sensible as we don't want to hang the application due to a slow client. But this means at some point one of the send queues will fill up and new messages will be discarded.

But it can: http://stackoverflow.com/a/15821036. So what can a subscriber do if it detects a missing message?

That's up to the application. Either stop with a fatal error, or re-start/re-sync. An event application usually does some synchronization at start-up, then starts processing incremental changes. After getting out of sync it can repeat the process. The important thing is being able to detect it and thus act on it.

This needs mention in doc/release-notes.md as well as doc/zmq.md (there, don't forget to specify the size and endianness of the sequence number), I think it is ready for merge otherwise.

Fixed @MarcoFalke nit.
Mentioned the change in doc/release-notes.md as well as in doc/zmq.md

@MarcoFalke: fixed nits.

Here's a slight simplification of the code - gets rid of some duplication, as well as the _keepalive change:
laanwj@fb843df

@laanwj: Thanks. Much better. Stolen fb843df, squashed and force pushed.

The way we are using ZMQ, the sender will never block. This is sensible as we don't want to hang the application due to a slow client.

Could we make that parameterizable?
In certain cases I'd rather have message reliability than having the node to be guaranteed to be running in real time.

In certain cases I'd rather have message reliability than having the node to be guaranteed to be running in real time.

Well notifications can be lost, through zmq or otherwise, for example if bitcoind needs to be restarted, or a myriad of other circumstances not under your control. At least you can detect it now:

• sequence number goes to 0 (and wasn't at 0xffffffff) -> bitcoind restarted
• sequence number skips a beat -> send buffer was full

Your application needs an (application dependent) way to resync anyhow. This is better than pretending to guarantee something.