Skip to content

Native Descriptor Wallets using DescriptorScriptPubKeyMan#16528

Merged
meshcollider merged 43 commits intobitcoin:masterfrom
achow101:wallet-of-the-glorious-future
Apr 27, 2020
Merged

Native Descriptor Wallets using DescriptorScriptPubKeyMan#16528
meshcollider merged 43 commits intobitcoin:masterfrom
achow101:wallet-of-the-glorious-future

Conversation

@achow101
Copy link
Member

@achow101 achow101 commented Aug 2, 2019

Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using createwallet.

Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each.

Descriptors can also be imported with a new importdescriptors RPC.

Native descriptor wallets use the ScriptPubKeyMan interface introduced in #16341 to add a DescriptorScriptPubKeyMan. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, DescriptorScriptPubKeyMan does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with disable_private_keys needs to be used for watchonly things.

A --descriptor option was added to some tests (wallet_basic.py, wallet_encryption.py, wallet_keypool.py, wallet_keypool_topup.py, and wallet_labels.py) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (importprivkey, importpubkey, importaddress, importmulti, addmultisigaddress, dumpprivkey, dumpwallet, importwallet, and sethdseed).

@fanquake fanquake added the Wallet label Aug 2, 2019
@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch from acbd19c to 6f8afc0 Compare August 2, 2019 02:06
@DrahtBot
Copy link
Contributor

DrahtBot commented Aug 2, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@meshcollider
Copy link
Contributor

Strong Concept ACK, this should be much nicer now it is preceded by the rework.

@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch 2 times, most recently from ef59fb7 to 90a631f Compare August 2, 2019 16:23
@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch 2 times, most recently from ef63001 to 755c4d1 Compare August 2, 2019 23:41
@Sjors
Copy link
Member

Sjors commented Aug 3, 2019

Approach ACK. It looks pretty straight forward and thanks to The Box feels cleaner than the previous attempt.

I suggest that, after a bit more progress on #16341, we review this in parallel. That ensures that we don't mess up the division of labour between ScriptPubKeyMan and its subclasses.

Also concept ACK on using BIP44/49/84 for new descriptor wallets. That may need some discussion, because it means individual addresses are no longer hardened.

Some issues I found perusing the commits:

  • when you restart bitcoind and load a (watch-only) wallet the keypool is empty and getnewaddress no longer works
  • importdescriptors should no longer require a range argument (also I would prefer a singular RPC nvm that makes rescan slow)
  • when calling getnewaddress with an address type for which the wallet misses a descriptor, it returns a blank error message:
  • can you give each ScriptPubKeyManager it's own file?
  • deleting GetMetadata and Upgrade inside Introduce WalletDescriptor; should have its own commit?
  • ScriptPubKeyMap could be introduced in Implement IsMine instead of the earlier commit. Would it make sense to move this into the Descriptor class? How are infinite range descriptors handled, expanded and cached keypoolsize items at a time?
  • Am I reading this wrong or is SetupGeneration creating a fresh seed for each descriptor?
  • I'm not a fan of determining the output type in an indirect manner by expanding the descriptor Optional<OutputType> out_script_type = DetermineOutputType(scripts_temp[0], out_keys);; that information is already encoded in the descriptor. Shameless plug for Descriptor: add GetAddressType() and IsSegWit() #15590.

@hugohn
Copy link
Contributor

hugohn commented Aug 12, 2019

Concept ACK.

@achow101 I just noticed that both the existing deriveaddressses & importmulti commands treat descriptor [range_start, range_end] as inclusive, which is consistent with the common understanding of the notation []. Should we update the new importdescriptors command to do the same (make range_end inclusive)?

https://github.com/bitcoin/bitcoin/blob/master/src/rpc/misc.cpp#L215
https://github.com/bitcoin/bitcoin/blob/master/src/wallet/rpcdump.cpp#L1122

@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch from 755c4d1 to d777554 Compare August 12, 2019 16:19
@achow101
Copy link
Member Author

Should we update the new importdescriptors command to do the same (make range_end inclusive)?

I guess so. Latest pushed should do that.

@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch from 693c8cc to 84fb7d7 Compare August 14, 2019 22:57
@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch 3 times, most recently from 655ea88 to a208faa Compare September 4, 2019 18:19
@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch from a208faa to f581c8c Compare September 4, 2019 22:54
@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch 2 times, most recently from f8edc40 to 69ad25c Compare September 18, 2019 03:38
@Sjors Sjors mentioned this pull request Sep 26, 2019
2 tasks
@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch from 69ad25c to 98b7838 Compare October 10, 2019 05:57
@Sjors
Copy link
Member

Sjors commented Oct 10, 2019

Due to your latest change upstream, this now complains: scriptpubkeyman.h:516:10: warning: 'CheckDecryptionKey' overrides a member function but is not marked 'override'

@achow101 achow101 force-pushed the wallet-of-the-glorious-future branch from 98b7838 to d80de7d Compare October 10, 2019 19:13
achow101 and others added 11 commits April 23, 2020 13:59
When a CWallet doesn't have a ScriptPubKeyMan for the requested type
in GetNewDestination, give a meaningful error. Also handle this in
Qt which did not do anything with errors.
RPCOverloadWrapper overloads some deprecated or disabled RPCs with
an implementation using other RPCs to avoid having a ton of code churn
around replacing those RPCs.
Adds a --descriptors option globally to the test framework. This will
make the test create and use descriptor wallets. However some tests may
not work with this.

Some tests are modified to work with --descriptors and run with that
option in test_runer:
* wallet_basic.py
* wallet_encryption.py
* wallet_keypool.py
* wallet_keypool_topup.py
* wallet_labels.py
* wallet_avoidreuse.py
@achow101
Copy link
Member Author

Addressed @jonatack's comments and rebased as requested by @Sjors

@Sjors
Copy link
Member

Sjors commented Apr 23, 2020

utACK 223588b (rebased, nits addressed)

That's intended. Not all tests have been reworked to work with descriptor wallets. wallet_balance.py is one of those tests that need some modifications.

I might be seeing two different issues. When I run the full test suite that particular test sometimes fails with [node 0] bitcoind exited with status 1 during initialization, but not every time, and it also happens with other tests, including non-wallet ones. And given that Travis macOS passes, I think it's unrelated to this PR and maybe a (new) dev setup problem.

@jonatack
Copy link
Member

Code review re-ACK 223588b.

Rebuilt, re-ran all tests, bitcoind and a few importdescriptors rpc commands as a sanity check. I did not test the GUI changes yet.

@fjahr
Copy link
Contributor

fjahr commented Apr 24, 2020

re-ACK 223588b

Changes were only rebase and addressing nits. FWIW, I did not see any failures from wallet_balance.py.

@instagibbs
Copy link
Member

instagibbs commented Apr 24, 2020

light re-ACK 223588b

Read carefully through the descriptor-specific tests one more time, as well as the discussion since my last review. Admittedly a light re-review. Some more advanced use-cases may require additional tooling later and that's ok.

Copy link
Contributor

@meshcollider meshcollider left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK 223588b

Only lightly looked at the functional tests, but the actual code looks great. Thanks for carrying this through achow101. Great to have this in so early in the 0.21 release cycle too.

throw std::runtime_error(std::string(__func__) + ": writing descriptor failed");
}

// TopUp
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Helpful comment 😅

KeyMap m_map_keys GUARDED_BY(cs_desc_man);
CryptedKeyMap m_map_crypted_keys GUARDED_BY(cs_desc_man);

bool SetCrypted();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be here should it

@practicalswift
Copy link
Contributor

People interested in this recently merged PR might be interested in reviewing the small follow-up PR #18782 ("wallet: Make sure no DescriptorScriptPubKeyMan or WalletDescriptor members are left uninitialized after construction") :)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.