Save 7% total memory usage by using pointers as keys in mapNextTx#7991
Closed
kazcw wants to merge 3 commits intobitcoin:masterfrom
Closed
Save 7% total memory usage by using pointers as keys in mapNextTx#7991kazcw wants to merge 3 commits intobitcoin:masterfrom
kazcw wants to merge 3 commits intobitcoin:masterfrom
Conversation
Saves about 7% of total memory usage (halves the size of each entry in mapNextTx, which comprised about 15% of memory usage in overnight test run). Since the mempool is DynamicUsage-regulated, this will translate to a larger mempool in the same amount of space.
Contributor
|
Any reason for close? |
Contributor
Author
|
I'm exploring an alternative solution; I'll PR one or the other soon. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I profiled bitcoind's memory usage and noticed that the tx hashes as keys in
mapNextTxare taking a large amount of space, but they're redundant since the corresponding value has a pointer to theCTransactionthat has the hash in it. On a system with 64-bit pointers, replacing these hashes with pointers or references would, once the mempool warms up, save about 7% of total memory usage.I explored different ways of doing this.
I ruled out
std::reference_wrapper(possibly the most straightforward way of using references as keys in a map), because it's a big footgun and also the syntax for using it would be weird; you'd have tostatic_cast<COutpoint&>(it->first)to get the "value" of a key from an iterator.The simplest approach I found was to use a
map<const COutPoint*, CInpoint, DereferencingComparator<const COutPoint*>>-- but this also required strange syntax:map.find(&outpoint)is misleading, since the outpoints will be compared as values.The implementation here is a refinement of the
DereferencingComparatorapproach: I createdrefmapas a simple proxy to a map that has pointers as keys and uses theDereferencingComparator;refmappasses through all methods except those that normally take a comparison value, which it takes by const reference and gives the address to the underlying map. This allows a map to use pointers as keys, and: