Merged
Conversation
Also implement ConvertInvalidTimeKeys to convert old entries. We can remove this later when we know that most MNs have run this code on testnet. The way we stored the time field in the past lead to CleanupOldRecoveredSigs iterating the keys in a strange order, causing no deletion at all and the LLMQ DB filling up.
This avoids RAM filling up and OOM getting triggered.
Instead of only deleting when the corresponding recovered sig is deleted. It sometimes happens that a masternode votes on something but a recovered sig is never created, which leaves us with a vote that will never be deleted.
Member
PastaPastaPasta
left a comment
There was a problem hiding this comment.
a suggestion and a question for you
| CRecoveredSigsDb::CRecoveredSigsDb(CDBWrapper& _db) : | ||
| db(_db) | ||
| { | ||
| if (Params().NetworkIDString() == CBaseChainParams::TESTNET) { |
Member
There was a problem hiding this comment.
Is there a reason why this only affects testnet?
Author
There was a problem hiding this comment.
Reason is that only testnet currently contains any of these values in the DB. Mainnet doesn't have these keys/values, even if you already run v14 on mainnet. I'd like to avoid having this rs_upgraded key to appear on mainnet. Also, I'd like to remove this upgrade code ASAP.
Member
There was a problem hiding this comment.
Sounds good, that's what I presumed
| hasSigForSessionCache.erase(signHash); | ||
| hasSigForHashCache.erase(recSig.GetHash()); | ||
|
|
||
| if (batch.SizeEstimate() >= (1 << 24)) { |
Member
There was a problem hiding this comment.
I'm not personally a big fan of conveying numbers like this but it's fine...
Co-Authored-By: PastaPastaPasta <[email protected]>
Author
|
Applied minor code suggestions. Need re-ACK |
MIPPL
pushed a commit
to biblepay/biblepay
that referenced
this pull request
Jun 16, 2019
* commit '05adda99fe09f9f6d99ce09e22ed89be3ddfcd27': (530 commits) Update release notes with latest commits (dashpay#2958) Only require valid collaterals for votes and triggers (dashpay#2947) (dashpay#2957) [v0.14.0.x] Fix off-by-one error in InstantSend mining info removal when disconnecting blocks (dashpay#2951) [v0.14.0.x] bump version to 0.14.0.1 and prepare release notes (dashpay#2952) Update release notes v14 (dashpay#2927) Set CLIENT_VERSION_IS_RELEASE to true (dashpay#2926) Update help text via gen-manpages.sh (dashpay#2929) 0.14 release notes draft (dashpay#2896) Fix duplicate `-instantsendnotify` invocation (dashpay#2925) Add blocks conflicting with ChainLocks to block index (dashpay#2923) Skip processing in SyncTransaction when chain is not synced yet (dashpay#2920) Set DIP0008 mainnet activation params (dashpay#2915) [0.14] Bump chainparams (dashpay#2910) Fix db leaks in LLMQ db (dashpay#2914) Fall back to ReadBlockFromDisk when blockTxs is not filled yet (dashpay#2908) Bump "keepOldConnections" by one for all LLMQ types (dashpay#2909) Print inputs on which we voted and quorums used for signing (dashpay#2907) Implement integration tests for DKG error handling (dashpay#2905) Implement zmq notifications for chainlocked blocks (dashpay#2899) Properly handle conflicts between ChainLocks and InstantSend (dashpay#2904) ... # Conflicts: # .travis.yml # COPYING # README.md # biblepay-docs/protocol-documentation.md # ci/matrix.sh # configure.ac # contrib/debian/examples/biblepay.conf # contrib/gitian-descriptors/gitian-linux.yml # contrib/gitian-descriptors/gitian-osx.yml # contrib/gitian-descriptors/gitian-win.yml # doc/README.md # doc/README_windows.txt # doc/dnsseed-policy.md # doc/files.md # doc/guide-startmany.md # doc/man/biblepay-cli.1 # doc/man/biblepay-qt.1 # doc/man/biblepay-tx.1 # doc/man/biblepayd.1 # doc/masternode-budget.md # doc/masternode_conf.md # doc/release-notes.md # qa/pull-tester/rpc-tests.py # qa/rpc-tests/autoix-mempool.py # qa/rpc-tests/dip3-deterministicmns.py # qa/rpc-tests/fundrawtransaction.py # qa/rpc-tests/maxblocksinflight.py # qa/rpc-tests/p2p-acceptblock.py # qa/rpc-tests/test_framework/comptool.py # qa/rpc-tests/test_framework/mininode.py # qa/rpc-tests/test_framework/test_framework.py # qa/rpc-tests/test_framework/util.py # qa/rpc-tests/wallet-hd.py # share/setup.nsi.in # src/Makefile.am # src/Makefile.bench.include # src/Makefile.qt.include # src/Makefile.qttest.include # src/Makefile.test.include # src/activemasternode.cpp # src/activemasternode.h # src/biblepayd.cpp # src/cachemap.h # src/cachemultimap.h # src/chainparams.cpp # src/clientversion.h # src/dsnotificationinterface.cpp # src/evo/deterministicmns.cpp # src/evo/providertx.cpp # src/evo/simplifiedmns.cpp # src/governance-classes.cpp # src/governance-classes.h # src/governance-object.cpp # src/governance-object.h # src/governance-vote.cpp # src/governance.cpp # src/governance.h # src/hash.h # src/hdchain.cpp # src/hdchain.h # src/init.cpp # src/instantx.cpp # src/instantx.h # src/keepass.cpp # src/keepass.h # src/llmq/quorums_blockprocessor.cpp # src/llmq/quorums_dummydkg.cpp # src/llmq/quorums_dummydkg.h # src/llmq/quorums_init.cpp # src/llmq/quorums_init.h # src/llmq/quorums_utils.cpp # src/llmq/quorums_utils.h # src/masternode-payments.cpp # src/masternode-payments.h # src/masternode-sync.cpp # src/masternode-sync.h # src/masternode.cpp # src/masternode.h # src/masternodeconfig.cpp # src/masternodeconfig.h # src/masternodeman.cpp # src/masternodeman.h # src/messagesigner.cpp # src/messagesigner.h # src/miner.cpp # src/net.cpp # src/net_processing.cpp # src/netfulfilledman.cpp # src/netfulfilledman.h # src/privatesend-client.cpp # src/privatesend-client.h # src/privatesend-server.cpp # src/privatesend-server.h # src/privatesend.cpp # src/privatesend.h # src/qt/askpassphrasedialog.cpp # src/qt/biblepay.cpp # src/qt/bitcoinaddressvalidator.cpp # src/qt/bitcoingui.cpp # src/qt/bitcoinunits.cpp # src/qt/bitcoinunits.h # src/qt/clientmodel.cpp # src/qt/coincontroldialog.cpp # src/qt/dashstrings.cpp # src/qt/editaddressdialog.cpp # src/qt/forms/intro.ui # src/qt/forms/masternodelist.ui # src/qt/guiconstants.h # src/qt/guiutil.cpp # src/qt/intro.cpp # src/qt/locale/biblepay_ar.ts # src/qt/locale/biblepay_bg.ts # src/qt/locale/biblepay_de.ts # src/qt/locale/biblepay_en.ts # src/qt/locale/biblepay_es.ts # src/qt/locale/biblepay_fi.ts # src/qt/locale/biblepay_fr.ts # src/qt/locale/biblepay_it.ts # src/qt/locale/biblepay_ja.ts # src/qt/locale/biblepay_ko.ts # src/qt/locale/biblepay_nl.ts # src/qt/locale/biblepay_pl.ts # src/qt/locale/biblepay_pt.ts # src/qt/locale/biblepay_ru.ts # src/qt/locale/biblepay_sk.ts # src/qt/locale/biblepay_th.ts # src/qt/locale/biblepay_tr.ts # src/qt/locale/biblepay_vi.ts # src/qt/locale/biblepay_zh_CN.ts # src/qt/locale/biblepay_zh_TW.ts # src/qt/networkstyle.h # src/qt/openuridialog.cpp # src/qt/optionsdialog.cpp # src/qt/optionsmodel.cpp # src/qt/overviewpage.cpp # src/qt/res/images/light/about.png # src/qt/res/images/light/drkblue_walletFrame_bg.png # src/qt/rpcconsole.cpp # src/qt/sendcoinsdialog.cpp # src/qt/sendcoinsentry.cpp # src/qt/splashscreen.cpp # src/qt/transactiondesc.cpp # src/qt/transactionrecord.cpp # src/qt/utilitydialog.cpp # src/qt/walletmodel.cpp # src/rpc/blockchain.cpp # src/rpc/client.cpp # src/rpc/governance.cpp # src/rpc/masternode.cpp # src/rpc/mining.cpp # src/rpc/misc.cpp # src/rpc/net.cpp # src/rpc/rawtransaction.cpp # src/rpc/rpcevo.cpp # src/rpc/server.cpp # src/spork.cpp # src/spork.h # src/test/bip39_tests.cpp # src/test/cachemap_tests.cpp # src/test/cachemultimap_tests.cpp # src/test/governance_validators_tests.cpp # src/test/miner_tests.cpp # src/test/ratecheck_tests.cpp # src/test/test_biblepay.h # src/uint256.h # src/util.cpp # src/util.h # src/validation.cpp # src/validation.h # src/version.h # src/wallet/rpcdump.cpp # src/wallet/rpcwallet.cpp # src/wallet/wallet.cpp # src/wallet/wallet.h # src/wallet/walletdb.cpp
panleone
pushed a commit
to panleone/PIVX
that referenced
this pull request
Nov 14, 2024
* Store rs_t key time in big endian Also implement ConvertInvalidTimeKeys to convert old entries. We can remove this later when we know that most MNs have run this code on testnet. The way we stored the time field in the past lead to CleanupOldRecoveredSigs iterating the keys in a strange order, causing no deletion at all and the LLMQ DB filling up. * Write batch in CleanupOldRecoveredSigs when it gets too large This avoids RAM filling up and OOM getting triggered. * Keep track of when a vote was written to the DB and clean up after week Instead of only deleting when the corresponding recovered sig is deleted. It sometimes happens that a masternode votes on something but a recovered sig is never created, which leaves us with a vote that will never be deleted. * Apply suggestions from code review Co-Authored-By: PastaPastaPasta <[email protected]>
Fuzzbawls
added a commit
to PIVX-Project/PIVX
that referenced
this pull request
Nov 15, 2024
c7e2beb scripted-diff: Refactor llmq type consensus param names (dashpay#3093) (UdjinM6) af7bb99 Re-verify invalid IS sigs when the active quorum set rotated (dashpay#3052) (Alexander Block) be20a71 Remove recovered sigs from the LLMQ db when corresponding IS locks get confirmed (dashpay#3048) (Alexander Block) 802a933 Don't wake up select if it was already woken up (dashpay#2863) (Alexander Block) 436300d Disable optimistic send in PushMessage by default (dashpay#2859) (Alexander Block) 44ad484 Optimize LLMQs sending of sig shares (dashpay#2704) (Alexander Block) e01ad46 Fix db leaks in LLMQ db (dashpay#2914) (Alexander Block) d2a2d15 Print inputs on which we voted and quorums used for signing (dashpay#2907) (Alexander Block) dce46ad Bail out in few more places when blockchain is not synced yet (dashpay#2888) (UdjinM6) 965a4a7 Use lazy BLS signatures more often and don't always verify self-recovered sigs (dashpay#2860) (Alexander Block) f4a5a04 cherry pick dashpay#2889 (UdjinM6) Pull request description: each commit backports a different PR ACKs for top commit: c7e2beb Duddino: utACK c7e2beb Liquid369: utACK c7e2beb Tree-SHA512: 86425039d01990ff04eea95f5ea4bee4821eaa80c33bab147d5b9407589c799c8293c2da5d5faf10e7e66b65d568444318c87fc274f39b42d2b69b80abafb417
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There were 2 leaks in the LLMQ db which caused it to grow bigger and bigger. This PR fixes both. See individual commits for the actual fixes.