Merge settings one place instead of five places#15934
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
promag
left a comment
There was a problem hiding this comment.
I had a refactor (which I did't submit) that supported chaining ArgsManager. The idea was to support changing some args when calling some RPC, so a ArgsManager is created with the "overridden" args and passed thru. Is this something you are considering supporting or do you see a different approach?
Concept ACK.
19e84b7 to
1d543ad
Compare
|
re: #15934 (review) from promag
This change does make it easier to add new settings sources (with consistent handling of negated args and things), so it should be compatible with your idea and maybe helpful. Depending on the situation, I think having chained or scoped settings could be a good idea or not. I do think that in wallet code and application code generally it's good to get away from using key-value storage classes like |
|
Concept ACK |
jamesob
left a comment
There was a problem hiding this comment.
Tested ACK 1d543ad
Generated and reviewed the test output locally. Mucked around with various argument formulations using the following config file:
dbcache=100
[main]
dbcache=200
[test]
dbcache=300
and commandline invocations e.g.
./src/bitcoind -conf=$(pwd)/test.conf -dbcache=1000 -dbcache=500 | grep Usingto verify dbcache being set as expected.
This is a well-written change that cleans up a lot of gnarly, duplicated settings munging. It explicitly outlines surprising corner cases in existing behavior (with docs too), and makes reasoning about settings merge order easier. This change also introduces substantial test coverage to settings management (util::Settings).
After this is merged, adding a read-write settings file (whether it's JSON or something else) will be much easier.
src/util/system.cpp
Outdated
| // Weird behavior preserved for backwards compatibility: command line | ||
| // options with section prefixes are allowed but ignored. It would be | ||
| // better if these options triggered the IsArgKnown error below, or were | ||
| // actually used instead of silently ignored. |
There was a problem hiding this comment.
Thanks for the nice comment. Potentially out of scope: could we log warnings for this instead of silently ignoring?
There was a problem hiding this comment.
re: #15934 (comment)
Potentially out of scope: could we log warnings for this instead of silently ignoring?
I'm planning on making followup PRs that simplify and clean up all these "Weird behavior preserved for backwards compatibility" instances. I'd rather not add warnings in this PR partly because I'm disinclined to mix up behavior changes and refactoring changes in the same PR, but also because having some invalid options result in warnings and other invalid options result in errors seems even more strange and complicated than what the code does now.
| // test parses and merges settings, representing the results as strings that get | ||
| // compared against an expected hash. To debug, the result strings can be dumped | ||
| // to a file (see comments below). | ||
| BOOST_FIXTURE_TEST_CASE(Merge, MergeTestingSetup) |
There was a problem hiding this comment.
Cool test! The formatted output is really helpful. Encourage other reviewers to run and inspect with
SETTINGS_MERGE_TEST_OUT=results.txt ./src/test/test_bitcoin --run_test=settings_tests/Merge
ryanofsky
left a comment
There was a problem hiding this comment.
Updated 1d543ad -> 2dfeff1 (pr/mergeset.6 -> pr/mergeset.7), compare) with suggested changes.
Rebased 2dfeff1 -> 955c782 (pr/mergeset.7 -> pr/mergeset.8) to share common code with #15988.
src/util/system.cpp
Outdated
| // Weird behavior preserved for backwards compatibility: command line | ||
| // options with section prefixes are allowed but ignored. It would be | ||
| // better if these options triggered the IsArgKnown error below, or were | ||
| // actually used instead of silently ignored. |
There was a problem hiding this comment.
re: #15934 (comment)
Potentially out of scope: could we log warnings for this instead of silently ignoring?
I'm planning on making followup PRs that simplify and clean up all these "Weird behavior preserved for backwards compatibility" instances. I'd rather not add warnings in this PR partly because I'm disinclined to mix up behavior changes and refactoring changes in the same PR, but also because having some invalid options result in warnings and other invalid options result in errors seems even more strange and complicated than what the code does now.
|
re-tACK 955c782 based on the interdiff and running an abbreviated version of the testing above. |
|
Rebased 955c782 -> 14a6dfc (pr/mergeset.8 -> pr/mergeset.9) due to conflict with #16278 |
|
reACK 14a6dfc based on interdiff. Only change since |
|
ACK 083c954 I've reviewed that's new code is doing what it says to do. I think that weird behaviors are compatibility-maintained but at least if we have a regression they are well-documented and so we'll know where to look, plus as said by James this part of code is low-risk so it's worth moving forward. Great work Russ, and I'm concept ACK future PRs to remove/squeeze weird behaviors! |
| error += "-includeconf cannot be used from commandline; -includeconf=" + ic + "\n"; | ||
| } | ||
| return false; | ||
| bool success = true; |
There was a problem hiding this comment.
nit (please don't change in this PR. Leave for a follow-up, if at all): 'we clear it' in the comment above is inaccurate. If an -includeconf is found on the command line then we return false rather than silently removing the bad config. Change that comment to // Do not allow -includeconf from the command line (except for -noincludeconf)
In fact, I don't think we need the local success variable at all. You could just return false from the inner for loop. I don't think the user needs to be told about the multiple -includeconf command line arguments they've used. Just alerting about the first should be enough.
|
Thanks! Will leave PR at 083c954. Some notes for future followup:
|
|
Other potential future follow-ups:
( Line 970 in 8021392 Line 88 in 8021392 GetChainName() is called in ReadConfigFiles(), and so starting with -regtest -testnet results in an unhandled exception.
Here in master: Line 165 in 8021392 ArgsManager(). The class was only added to speed up rebuilding (#11862 (comment)), which doesn't seem like a good reason to keep it.
|
jamesob
left a comment
There was a problem hiding this comment.
ACK 083c954
Reviewed diff-of-diffs both on Github and locally. Only changes since my last ACK are those discussed above. Nice job persevering, @ryanofsky, and thanks to @jnewbery @ariard for good feedback.
| // precedence over early settings, but for backwards compatibility in | ||
| // the config file the precedence is reversed for all settings except | ||
| // chain name settings. | ||
| const bool reverse_precedence = (source == Source::CONFIG_FILE_NETWORK_SECTION || source == Source::CONFIG_FILE_DEFAULT_SECTION) && !get_chain_name; |
There was a problem hiding this comment.
These long lines are killin' me but I'll bite my tongue in the interest of a merge.
There was a problem hiding this comment.
Will fix with the other followups, but if it in case its useful https://greasyfork.org/en/scripts/18789-github-toggle-code-wrap seems to work well. (Found through https://github.com/StylishThemes/GitHub-code-wrap)
083c954 Add settings_tests (Russell Yanofsky) 7f40528 Deduplicate settings merge code (Russell Yanofsky) 9dcb952 Add util::Settings struct and helper functions. (Russell Yanofsky) e2e37cf Remove includeconf nested scope (Russell Yanofsky) 5a84aa8 Rename includeconf variables for clarity (Russell Yanofsky) dc8e1e7 Clarify emptyIncludeConf logic (Russell Yanofsky) Pull request description: This is a refactoring-only change that makes it easier to add a new settings source. This PR doesn't change behavior. The [`util_ArgsMerge`](https://github.com/bitcoin/bitcoin/blob/deb2327b435925c6a39ca654a79283b8eb6aeb86/src/test/util_tests.cpp#L626-L822) and [`util_ChainMerge`](https://github.com/bitcoin/bitcoin/blob/deb2327b435925c6a39ca654a79283b8eb6aeb86/src/test/util_tests.cpp#L843-L924) tests added in #15869 and #15988 were written specifically to confirm that ArgsManager settings are parsed, merged, and returned the same way before and after this change. This change: - Makes it easier to add new settings sources that can get merged with existing sources (see 70675c3 from #15935). - Separates parsing of settings from merging of settings, and deduplicates merging code so it doesn't happen five different places ([GetArg](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L221-L244), [GetNetBoolArg](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L255-L261), [GetArgs](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L460-L467), [IsArgNegated](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L482-L491), [GetUnsuitableSectionOnlyArgs](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L343-L352)) in inconsistent ways. - Documents and tests current strange merging behaviors, so they be cleaned up in the future if resulting code simplifications and UX improvements warrant loss of backwards compatibility. The newly documented behaviors are: command line [ignored arguments](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/system.cpp#L323-L326) and [more ignored arguments](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L67-L72), and config file [reverse precedence](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L61-L65), [inconsistently applied top-level settings](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L55-L59), and [zombie values](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L101-L108). The original motivation for this change was to make it easy to add a new persistent setting source without introducing more bugs and inconsistencies. Two commits building on top of this to add a persistent `-wallet` setting are pretty straightforward and show how the new code can be extended: * 70675c3 from #15935 – _Add \<datadir>/settings.json persistent settings storage_ * 04c80c4 from #15937 – _Add loadwallet and createwallet RPC load_on_startup options_ ACKs for top commit: ariard: ACK 083c954 jnewbery: ACK 083c954 jamesob: ACK 083c954 Tree-SHA512: 5d106746a44d64d3963c4ef3f4a2fa668a4bedcc9018d3ea12c86beae2fda48a0b036241665837f68685712366f70f2e1faba84d193fa1f456013503097b7659
Suggestion from John Newbery <[email protected]> in bitcoin#15934 (comment)
includeconf -> conf_file_names to_include -> conf_file_name include_config -> conf_file_stream Suggestion from John Newbery <[email protected]> in bitcoin#15934 (comment)
Easier to review ignoring whitespace Suggestion from John Newbery <[email protected]> in bitcoin#15934 (comment)
083c954 Add settings_tests (Russell Yanofsky) 7f40528 Deduplicate settings merge code (Russell Yanofsky) 9dcb952 Add util::Settings struct and helper functions. (Russell Yanofsky) e2e37cf Remove includeconf nested scope (Russell Yanofsky) 5a84aa8 Rename includeconf variables for clarity (Russell Yanofsky) dc8e1e7 Clarify emptyIncludeConf logic (Russell Yanofsky) Pull request description: This is a refactoring-only change that makes it easier to add a new settings source. This PR doesn't change behavior. The [`util_ArgsMerge`](https://github.com/bitcoin/bitcoin/blob/deb2327b435925c6a39ca654a79283b8eb6aeb86/src/test/util_tests.cpp#L626-L822) and [`util_ChainMerge`](https://github.com/bitcoin/bitcoin/blob/deb2327b435925c6a39ca654a79283b8eb6aeb86/src/test/util_tests.cpp#L843-L924) tests added in bitcoin#15869 and bitcoin#15988 were written specifically to confirm that ArgsManager settings are parsed, merged, and returned the same way before and after this change. This change: - Makes it easier to add new settings sources that can get merged with existing sources (see 70675c3 from bitcoin#15935). - Separates parsing of settings from merging of settings, and deduplicates merging code so it doesn't happen five different places ([GetArg](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L221-L244), [GetNetBoolArg](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L255-L261), [GetArgs](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L460-L467), [IsArgNegated](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L482-L491), [GetUnsuitableSectionOnlyArgs](https://github.com/bitcoin/bitcoin/blob/c459c5f70176928adcee4935813a2dbe7f4dbd51/src/util/system.cpp#L343-L352)) in inconsistent ways. - Documents and tests current strange merging behaviors, so they be cleaned up in the future if resulting code simplifications and UX improvements warrant loss of backwards compatibility. The newly documented behaviors are: command line [ignored arguments](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/system.cpp#L323-L326) and [more ignored arguments](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L67-L72), and config file [reverse precedence](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L61-L65), [inconsistently applied top-level settings](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L55-L59), and [zombie values](https://github.com/ryanofsky/bitcoin/blob/69d44f3cc75a68d404ca0e1ca2b4831fd2bac4bb/src/util/settings.cpp#L101-L108). The original motivation for this change was to make it easy to add a new persistent setting source without introducing more bugs and inconsistencies. Two commits building on top of this to add a persistent `-wallet` setting are pretty straightforward and show how the new code can be extended: * 70675c3 from bitcoin#15935 – _Add \<datadir>/settings.json persistent settings storage_ * 04c80c4 from bitcoin#15937 – _Add loadwallet and createwallet RPC load_on_startup options_ ACKs for top commit: ariard: ACK 083c954 jnewbery: ACK 083c954 jamesob: ACK 083c954 Tree-SHA512: 5d106746a44d64d3963c4ef3f4a2fa668a4bedcc9018d3ea12c86beae2fda48a0b036241665837f68685712366f70f2e1faba84d193fa1f456013503097b7659
Suggested by James O'Beirne <[email protected]> bitcoin#15934 (comment) This commit does not change behavior.
Suggested by Antoine Riard <[email protected]> bitcoin#15934 (comment) and John Newbery <[email protected]> bitcoin#15934 (comment) This commit does not change behavior.
Rename suggested by João Barbosa <[email protected]> bitcoin#16545 (comment) This also gets rid of ArgsManager::NONE constant, which was an implementation detail not meant to be used by ArgsManager callers. Finally this reverts a change from 7f40528 bitcoin#15934 adding "-" characters to argument names. Better for GetArgFlags to require "-" prefixes for consistency with other argument methods, and to be more efficient later when GetArg functions need to call GetArgFlags. This commit does not change behavior.
Suggested by John Newbery <[email protected]> bitcoin#15934 (comment) This commit does not change behavior.
Rename suggested by João Barbosa <[email protected]> bitcoin#16545 (comment) This also gets rid of ArgsManager::NONE constant, which was an implementation detail not meant to be used by ArgsManager callers. Finally this reverts a change from 7f40528 bitcoin#15934 adding "-" characters to argument names. Better for GetArgFlags to require "-" prefixes for consistency with other ArgsManager methods, and to be more efficient later when GetArg functions need to call GetArgFlags (bitcoin#16545) This commit does not change behavior.
Suggested by John Newbery <[email protected]> bitcoin#15934 (comment) This commit does not change behavior.
|
I've been working on some followups for this PR since it's been merged. #17473 has various refactoring cleanups suggested above, and #17508 is tracking cleanup of confusing settings behaviors. Going through all the comments in this issue, I think the only suggestions left not addressed by #17473 and #17508 are:
I don't have plans to work on these last few things |
|
Maybe tag them good_first_issue or up_to_grabs to let people find them |
Suggested by James O'Beirne <[email protected]> bitcoin#15934 (comment) This commit does not change behavior.
|
Hidden by GitHub, but the "Up for grabs" refers to #15934 (comment) |
This is a refactoring-only change that makes it easier to add a new settings source.
This PR doesn't change behavior. The
util_ArgsMergeandutil_ChainMergetests added in #15869 and #15988 were written specifically to confirm that ArgsManager settings are parsed, merged, and returned the same way before and after this change.This change:
The original motivation for this change was to make it easy to add a new persistent setting source without introducing more bugs and inconsistencies. Two commits building on top of this to add a persistent
-walletsetting are pretty straightforward and show how the new code can be extended: