utACK fe84334

Concept ACK

Original chainstate (up to height=430234).

2.4G    testbtc.olddb/chainstate

After converting using master (and shutting down immediately):

2017-06-07 12:54:30 Upgrading database...
2017-06-07 13:09:39 LoadBlockIndexDB: last block file = 628

4.5G    chainstate

Converting using this pull (and shutting down immediately):

2017-06-07 12:29:37 Upgrading database...
2017-06-07 12:41:50 LoadBlockIndexDB: last block file = 628

4.4G    testbtc.newdb/chainstate

In this test, it doesn't seem to make a significant difference.

I wonder what to do here. Without any forced compaction, it's likely that many nodes will remain with a database that still contains table files for the old rows for an extended period of time.

Also, needs rebase.

After converting up to block height=430234 (same dataset as last time), hardwiring shutdown immediately after the block database processing.

Tested with rebased version here: https://github.com/laanwj/bitcoin/tree/2017_07_sipa_onthefly

2.1G    chainstate/

With master:

4.3G    chainstate/

So it seems the negative result last time was a fluke. utACK.

I repeated a few times, and it seems that it's like a coin toss. Sometimes it ends up as 4.4G, sometimes as 2.1G.
When the former, every time after running bitcoind again, even terminating immediately after chainstate processing, it was at 2.1G after shutting down.
It might be that a cleanup is re-triggered when re-opening the database, or when closing it.

Rebased. I've changed the final compact request to be for the entire per-tx range of keys, rather than just the last group updated. Maybe that triggers compaction sooner. Or not.

If someone was running with master before this commit, how does their database ever get compacted?

@morcos leveldb does automatic background compaction.

@morcos leveldb does automatic background compaction.

when? I have nodes running for a week that have 5.8G chainstate despite recent repeated stopping and starting.

@morcos Yea, I have one two weeks now. Perhaps we should also have some general compaction thing to clean up existing databases too....

I got some strange testing results...current master:

• Convert time 26:30, final chainstate size 3.5G
• Convert time 28:05, final chainstate size 3.5G

This PR:

• Convert time: 16:51, final chainstate size 5.9G (down to 2.7 immediately upon restart)
• Convert time: 16:11, final chainstate size 5.9G (down to 2.7 immediately upon restart)

Either way, I dont care about the excess usage if its faster and gets cleared upon restart (might get cleared in the background, too, i just made bitcoind do a clean exit right after chain state gets loaded).

@morcos hmm, I guess its just done when tables fill up, it should /eventually/ happen, but may take a long time, I suppose.

@TheBlueMatt wow, that's some really long convert times
My convert times with or without this PR were on the order of between 4 mins and 7 mins with some variation.

utACK efeb273, I tested an earlier version of the patch which should be equivalent, however.