wallet: Migrate legacy wallets to descriptor wallets without requiring BDB#26596
wallet: Migrate legacy wallets to descriptor wallets without requiring BDB#26596glozow merged 6 commits intobitcoin:masterfrom
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
9d6ce8a to
03fdbaf
Compare
|
I think it would be easier to review if you made a separate PR for I'm pleasantly surprised at how small the implementation is. |
03fdbaf to
bb4ea93
Compare
|
I don't see how reimplementing BDB is better than just continuing to use BDB. |
bb4ea93 to
a8060af
Compare
The reimplementation is quite compact because it is for only reading BDB files, only the features that we use, and none of the actual DB system stuff. BDB itself contains a ton of code, a lot of which we don't use. Maintaining it as a dependency will eventually be more burdensome - we already need to patch it in order to compile in the depends system. This reimplementation does not carry those issues - it implements a format that isn't going to change, and it only depends on things that are already being used elsewhere in the codebase. |
I've opeend #26606 for just |
a8060af to
b29c0bf
Compare
b29c0bf to
cfc73d1
Compare
cfc73d1 to
1af95bf
Compare
1af95bf to
928b022
Compare
|
Rebased following #26606. Now ready for review. |
cbergqvist
left a comment
There was a problem hiding this comment.
Code reviewed e11fa4524723aebc0d879ee083c2d6d46849d89a.
Overall concept: Not fully convinced about the benefit of breaking out LegacyDataSPKM, seems to add complexity. Is the goal to delete LegacyScriptPubKeyMan in the future when ripping out BDB and only keep LegacyDataSPKM?
Why perform the inlining of IsMine() in 31e918afc3920a2f8aa367ad323f590b4856aa26 before the function is removed? Seems risky in case of other PRs modifying it prior to removal etc.
LegacyScriptPubKeyMan::NewKeyPool() only touches data from LegacyDataSPKM, should it be moved to that class as public (or protected and then exposed in the subclass?). (If LegacyScriptPubKeyMan is slated for removal I can understand you want to signal that method will no longer be needed). (Have not checked if there are other methods along the same lines).
Assuming LegacyDataSPKM name is kept, comment in src/script/signingprovider.h could be updated to refer to LegacyDataSPKM::setWatchOnly instead.
src/wallet/scriptpubkeyman.cpp
Outdated
There was a problem hiding this comment.
Worth adding a comment about the consequences of setting keypool_size=0? I'm guessing there are still cases where we generate some pubkeys despite having a zero pool size, otherwise you would have removed some of the code and the for-loop below?
There was a problem hiding this comment.
I don't think this needs to have any additional explanation. keypool_size only matters for ranged descriptors. Non-HD keys do not make ranged descriptors, so the value here is ignored. The loop below is for an entirely different set of descriptors and is unrelated to this.
There was a problem hiding this comment.
Haven't fully grokked the code yet, so I understand if you'd rather not answer my questions, but if you want:
I don't think this needs to have any additional explanation. keypool_size only matters for ranged descriptors. Non-HD keys do not make ranged descriptors, so the value here is ignored.
We're in an HD chain for-loop, how is this not involving HD keys?
Surely legacy wallets could be HD wallets.
The loop below is for an entirely different set of descriptors and is unrelated to this.
I'm referring to the loop over desc_spks which is using the pubkeys from the DescriptorScriptPubKeyMan that's just been created and comes from DescriptorScriptPubKeyMan::m_map_script_pub_keys which is filled out in DescriptorScriptPubKeyMan::TopUpWithDB() which is affected by m_keypool_size.
Would m_keypool_size already always be 0 before this change?
There was a problem hiding this comment.
The line this comment is on is in the one that goes over the non-HD keys. However, I see that you're confused about the similar line in the loop below that iterates the hd chains.
The keypool_size here is a runtime value that depends on the startup options. It's is set when the DescriptorScriptPubKeyman is instantiated on loading, and the value is never directly persisted to disk. It ends up written as part of range_end when range_end is updated during TopUp(), but range_end can also be set directly.
The value that does matter is the range_end stored in WalletDescriptor. When we are going over the hd chains, we set range_end to match the chain counter stored in the CHDChain that is being migrated. This results in a descriptor that will produce exactly the same scriptPubKeys as the original hd chain, including it's keypool. We are enforcing that the migrated descriptors produce exactly the same scriptPubKeys (no more, no less). Setting keypool_size could mean that TopUp() would generate more scriptPubKeys than we need/expect, which would cause migration to assert and crash.
There was a problem hiding this comment.
Thanks for taking the time to clarify this matter!
Yes. We are trying to remove the legacy wallet in its entirety, leaving only behind the absolute minimum necessary to perform migration.
No. The separation exists now to indicate which things can be removed at the end. |
src/wallet/scriptpubkeyman.h
Outdated
There was a problem hiding this comment.
| // Implements the full legacy wallet behavior | |
| class LegacyScriptPubKeyMan : public LegacyDataSPKM | |
| // Implements the full legacy wallet behavior | |
| // Slated for deletion once Berkeley DB library dependency is removed. | |
| class LegacyScriptPubKeyMan : public LegacyDataSPKM |
There was a problem hiding this comment.
I don't think it's necessary to have a comment saying such.
There was a problem hiding this comment.
If the code is unable to communicate why it is like it is, it's good to explain the otherwise surprising state of it. It would hopefully limit the amount of questions like my prior one from #26596 (review):
LegacyScriptPubKeyMan::NewKeyPool()only touches data fromLegacyDataSPKM, should it be moved to that class as public (or protected and then exposed in the subclass?).
But I guess you're fairly confident it will be deleted in time for the next release anyway.
src/wallet/scriptpubkeyman.cpp
Outdated
There was a problem hiding this comment.
Haven't fully grokked the code yet, so I understand if you'd rather not answer my questions, but if you want:
I don't think this needs to have any additional explanation. keypool_size only matters for ranged descriptors. Non-HD keys do not make ranged descriptors, so the value here is ignored.
We're in an HD chain for-loop, how is this not involving HD keys?
Surely legacy wallets could be HD wallets.
The loop below is for an entirely different set of descriptors and is unrelated to this.
I'm referring to the loop over desc_spks which is using the pubkeys from the DescriptorScriptPubKeyMan that's just been created and comes from DescriptorScriptPubKeyMan::m_map_script_pub_keys which is filled out in DescriptorScriptPubKeyMan::TopUpWithDB() which is affected by m_keypool_size.
Would m_keypool_size already always be 0 before this change?
cbergqvist
left a comment
There was a problem hiding this comment.
ACK a8f8a2d5d0e2328cbe2e569c7e12f653d18f456d
(Ran tests on 31632d18ea122def82820770f14d9aa009726114 but only comments changed since then).
Functional tests including --extended passed (a few automatically skipped).
make check passed.
| if (!IsBDBFile(BDBDataFile(*wallet_path))) { | ||
| return util::Error{_("Error: This wallet is already a descriptor wallet")}; | ||
| } |
There was a problem hiding this comment.
in commit f732495: now that the legacy-wallet-check is done without loading the wallet, is the check for the descriptor flag below still needed? (can a BDB wallet ever have this flag set?)
// Before anything else, check if there is something to migrate.
if (local_wallet->IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS)) {
if (was_loaded) {
reload_wallet(local_wallet);
}
return util::Error{_("Error: This wallet is already a descriptor wallet")};
}
There was a problem hiding this comment.
It is theoretically possible to have a descriptor wallet using BDB under 2 scenarios, however this is still an unsupported configuration:
- The descriptor wallet was created on an unreleased version after Native Descriptor Wallets using DescriptorScriptPubKeyMan #16528 but before wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets #19077 (there was a period of about 6 months where this could have happened)
- The user manipulated the database using external tooling where they created a BDB file and manually inserted descriptor wallet records
src/wallet/scriptpubkeyman.cpp
Outdated
There was a problem hiding this comment.
Not really an issue but this works because TopUp sets the descriptor range_end to 1 on the first run when the descriptor is not ranged.
As this would fail if we ever change that, what if we set the range properly in WalletDescriptor constructor?
E.g.
WalletDescriptor w_desc(std::move(desc), creation_time, /*range_start=*/0, /*range_end=*/1, /*next_index=*/0)Also, this change doesn't seem to be related to c653f4fdbfe06 description? --> nvm about this last point, theStack just made me notice that this is because m_keypool_size is not accesible from LegacyDataSPKM.
furszy
left a comment
There was a problem hiding this comment.
Just a small comment; you don't have to take it.
I believe this PR could get merged quite fast if you leave 89503710386f52d9162f67fcd707eceffa954faa for a follow-up and make IsMine compatible with LegacyDataSPKM in this PR.
I'm suggesting this because this PR starts testing the newly introduced BDB reader class and allows people to use it, which is great since we want it polished for the next release. The IsMine removal is also nice but I wouldn't miss a deadline for it.
|
Still trying to fully grasp the
|
furszy
left a comment
There was a problem hiding this comment.
- To my understanding, the class
LegacyDataSPKMis supposed to never write to the database (asBerkeleyRODatabaseobviously doesn't support that), butLegacyDataSPKM::CheckDecryptionKeycontains a code-path for that (for a necessary rewrite):
https://github.com/achow101/bitcoin/blob/a8f8a2d5d0e2328cbe2e569c7e12f653d18f456d/src/wallet/scriptpubkeyman.cpp#L254-L255
ShouldCheckDecryptionKeystay inLegacyScriptPubKeyMan?
It isn't really an issue because CheckDecryptionKey does not verify the outcome of the write call (which would be "successful" because the read-only bdb class return true for all write operations). But.. for the sake of correctness I see two possible paths to follow:
-
As this write is done exclusively on wallets with no checksum on crypted keys, we could move the upgrade code to the loading code in
walletdb.cpp; removingfDecryptionThoroughlyCheckedentirely and requiring such ancient wallets to provide the passphrase during load to perform the upgrade (adding a "passphrase" arg to theloadwalletRPC command). -
Introduce a new
WalletBatch::IsReadOnly()method to avoid calling the write insideCheckDecryptionKey.
The first approach will make the legacy wallet code simpler while the second one is a minimal change.
Moved to #30328
I think it needs to be in the next release as it is a critical part of migration. I would prefer to have it in a release that people can use before deleting the legacy wallet.
Updated
Fixed
No, we should not be migrating a wallet if we cannot decrypt the keys. |
furszy
left a comment
There was a problem hiding this comment.
Strange CI https://github.com/bitcoin/bitcoin/actions/runs/9649050504/job/26611651920?pr=26596 error.. it doesn't seem to be related. The "salvage" command opens the wallet in DatabaseFormat::BERKELEY format.
When we reopen the wallet to do the migration, instead of opening using BDB, open it using the BerkeleyRO implementation.
|
Now that the Other than that, code review ACK 9700c21. |
cbergqvist
left a comment
There was a problem hiding this comment.
git range-diff master a8f8a2d 9700c217
Agree with @furszy on reordering the commits to make it possible to retain the 3 asserts in MigrateToDescriptor().
In order to load the necessary data for migrating a legacy wallet without the full LegacyScriptPubKeyMan, move the data storage and loading components to LegacyDataSPKM. LegacyScriptPubKeyMan now subclasses that.
IsMine is necessary for migration. It should be inlined with migration when the legacy wallet is removed.
In SetupLegacyScriptPubKeyMan, a base LegacyDataSPKM will be created if the database has the format "bdb_ro" (i.e. the wallet was opened only for migration purposes). All of the loading functions are now called with a LegacyDataSPKM object instead of LegacyScriptPubKeyMan.
The migration process reloads the wallet after all failures. This commit tests the behavior by trying to obtain a new address after a decryption failure during migration.
|
Reordered the commits and preserved the |
cbergqvist
left a comment
There was a problem hiding this comment.
ACK 8ce3739
git range-diff 9700c217~5..9700c217 8ce3739~5..8ce3739 - not the easiest read, also used GH compare.
Passed make check & test/functional/test_runner.py including p2p_handshake.py which failed on CI.
Only one non-critical reservation: #26596 (comment)
| LegacyDataSPKM* CWallet::GetLegacyDataSPKM() const | ||
| { | ||
| if (IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS)) { | ||
| return nullptr; | ||
| } | ||
| auto it = m_internal_spk_managers.find(OutputType::LEGACY); | ||
| if (it == m_internal_spk_managers.end()) return nullptr; | ||
| return dynamic_cast<LegacyDataSPKM*>(it->second); | ||
| } |
There was a problem hiding this comment.
nit: was about to suggest to deduplicate shared code between this method and ::GetLegacyScriptPubKeyMan above, but since the latter is removed soon anyway (in PR #28710, commit 5652a9c#diff-1f2db0e4d5c12d109c7f0962333c245b49b696cb39ff432da048e9d6c08944d8), it's probably not worth it.
There was a problem hiding this comment.
If you re-touch it again. Could:
LegacyDataSPKM* CWallet::GetLegacyDataSPKM() const
{
if (IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS)) { return nullptr; }
return dynamic_cast<LegacyDataSPKM*>(GetScriptPubKeyMan(OutputType::LEGACY, /*internal=*/true));
}| LegacyDataSPKM* CWallet::GetLegacyDataSPKM() const | ||
| { | ||
| if (IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS)) { | ||
| return nullptr; | ||
| } | ||
| auto it = m_internal_spk_managers.find(OutputType::LEGACY); | ||
| if (it == m_internal_spk_managers.end()) return nullptr; | ||
| return dynamic_cast<LegacyDataSPKM*>(it->second); | ||
| } |
There was a problem hiding this comment.
If you re-touch it again. Could:
LegacyDataSPKM* CWallet::GetLegacyDataSPKM() const
{
if (IsWalletFlagSet(WALLET_FLAG_DESCRIPTORS)) { return nullptr; }
return dynamic_cast<LegacyDataSPKM*>(GetScriptPubKeyMan(OutputType::LEGACY, /*internal=*/true));
}| void LegacyScriptPubKeyMan::LoadKeyMetadata(const CKeyID& keyID, const CKeyMetadata& meta) | ||
| { | ||
| LOCK(cs_KeyStore); | ||
| LegacyDataSPKM::LoadKeyMetadata(keyID, meta); | ||
| UpdateTimeFirstKey(meta.nCreateTime); | ||
| mapKeyMetadata[keyID] = meta; | ||
| } |
There was a problem hiding this comment.
In 7461d0c:
At this point, just a non-blocking nit but you are locking cs_KeyStore twice. One here, and another one inside LegacyDataSPKM::LoadKeyMetadata.
Same for LoadScriptMetadata.
| bool LegacyDataSPKM::AddKeyPubKeyInner(const CKey& key, const CPubKey& pubkey) | ||
| { | ||
| LOCK(cs_KeyStore); | ||
| return FillableSigningProvider::AddKeyPubKey(key, pubkey); |
There was a problem hiding this comment.
In 7461d0c:
FillableSigningProvider::AddKeyPubKey already locks cs_Keystore internally.
#26606 introduced
BerkeleyRODatabasewhich is an independent parser for BDB files. This PR uses this in legacy wallet migration so that migration will continue to work once the legacy wallet and BDB are removed.LegacyDataSPKMis introduced to have the minimum data and functions necessary for a legacy wallet to be loaded for migration.