Askrene: scale with many layers#8760
Conversation
|
Nice. I would like to see a benchmark to measure the time we spend fetching layers. UPD: I've noticed that the linked list of layers is worse than the hashtable adding elements because askrene checks if the layer exists before creating a new one, making askrene-create-layer behave like |
This means it has to do a lookup and an insert. Signed-off-by: Rusty Russell <[email protected]>
Simple bench. Before: Creating 20,000 layers: 20 seconds Creating 50,000 layers: 107 seconds After: Creating 20,000 layers: 7 seconds Creating 50,000 layers: 15 seconds Creating 100,000 layers: 29 seconds Signed-off-by: Rusty Russell <[email protected]>
…ers on startup. We used non-persistent layers before, but what if we save to the datastore and restore? This takes it from 29 to 45 seconds. Signed-off-by: Rusty Russell <[email protected]>
rustyrussell
force-pushed
the
askrene-many-layers
branch
from
f499188 to
dfc5d16
Compare
I re-ran with a rebase: it's halved. Technically we could send all the commands at once rather than waiting for the responses, but when I do that with futures in Python, the pytest starts dominating the CPU and the result is actually slower :( |
2 participants
@Lagrang3 pointed out that our single linked list of layers will become a bottleneck, so this contains a quick benchmark for many layers, and replaces the linked list with a hashtable.
Changelog-None