feat(wallet): try batching multiple wallet db operations when possible, avoid wasting cpu cycles in AddToWalletIfInvolvingMe#5452
Merged
UdjinM6 merged 2 commits intodashpay:developfrom Jun 28, 2023
Conversation
move findBlock out of the loop
5 tasks
PastaPastaPasta
approved these changes
Jun 28, 2023
Member
PastaPastaPasta
left a comment
PastaPastaPasta
left a comment
There was a problem hiding this comment.
utACK for squash merge
UdjinM6
added a commit
that referenced
this pull request
Jun 28, 2023
…pdates for huge notification queues (#5453) ## Issue being fixed or feature implemented It's super slow for wallets with 100.000s of txes to process lots of notifications produced by rescan. Skip them all and simply refresh the whole wallet instead. In my case (500k+ txes testnet wallet) gui update after `rescanblockchain` time is down from _forever_ to ~30 seconds. Same for `wipewallettxes true` (#5451 ). Gui update after `wipewallettxes`/`wipewallettxes false` is instant (cause there are no txes anymore) vs _forever_ before the patch. ## What was done? refresh the whole wallet when notification queue is above 10K operations actual changes (ignoring whitespaces): d013cb4?w=1 ## How Has This Been Tested? running on top of #5451 and #5452 , wiping and rescanning w/ and w/out this patch. ## Breaking Changes should be none ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
UdjinM6
added a commit
to UdjinM6/dash
that referenced
this pull request
Jul 25, 2023
…pdates for huge notification queues (dashpay#5453) ## Issue being fixed or feature implemented It's super slow for wallets with 100.000s of txes to process lots of notifications produced by rescan. Skip them all and simply refresh the whole wallet instead. In my case (500k+ txes testnet wallet) gui update after `rescanblockchain` time is down from _forever_ to ~30 seconds. Same for `wipewallettxes true` (dashpay#5451 ). Gui update after `wipewallettxes`/`wipewallettxes false` is instant (cause there are no txes anymore) vs _forever_ before the patch. ## What was done? refresh the whole wallet when notification queue is above 10K operations actual changes (ignoring whitespaces): dashpay@d013cb4?w=1 ## How Has This Been Tested? running on top of dashpay#5451 and dashpay#5452 , wiping and rescanning w/ and w/out this patch. ## Breaking Changes should be none ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
It's super slow for wallets with 100.000s of keys and txes to reindex and to rescan. Batching multiple operations fixes it. In my case (300K+ keys and 500k+ txes testnet wallet)
rescanblockchaintime is down from 6+ hours to ~10 minutes.Re-calculating
block_timeover and over again inside of the loop inAddToWalletIfInvolvingMeis wasteful, move it out.What was done?
batch what's possible, optimize
AddToWalletIfInvolvingMeHow Has This Been Tested?
running on top of #5451 , wiping and rescanning w/ and w/out this patch.
Breaking Changes
should be none
Checklist: