feat(rocksdb): tune LargeSmt profile and config#1947
Open
huitseeker wants to merge 3 commits intonextfrom
Open
feat(rocksdb): tune LargeSmt profile and config#1947huitseeker wants to merge 3 commits intonextfrom
huitseeker wants to merge 3 commits intonextfrom
Conversation
b552c83 to
c6ae78e
Compare
Tune the default LargeSmt RocksDB profile, expose tuning and memory-budget controls, restore durability mode selection, and keep config-builder coverage aligned with the backend changes. The default profile now uses cached index/filter blocks, 16 KiB block-based tables, per-column-family Bloom filter settings, and bottommost ZSTD compression for SMT data while keeping metadata uncompressed. The store startup path continues to use the existing cache-size and max-open-files knobs, and now also carries durability selection without widening the operational surface further than needed. Benchmark signal from prior account_tree Criterion quick runs supports carrying these defaults: the block-table profile change improved vanilla insertion/100 from 101.348 ms to 87.024 ms (-14.13%), with_history insertion/10 from 89.361 ms to 80.927 ms (-9.44%), and with_history insertion/500 from 109.341 ms to 95.542 ms (-12.62%). The bottommost ZSTD change improved vanilla insertion/1 from 84.445 ms to 73.982 ms (-12.39%, significant). Access-path deltas in those quick runs were mostly within Criterion's noise band, so this change is justified as a write-heavy default-profile improvement rather than a general across-the-board speedup.
bb578e9 to
e89b6a6
Compare
Collaborator
Mirko-von-Leipzig
left a comment
There was a problem hiding this comment.
I'm not entirely sure how to review something like this. Beyond saying that the code still compiles so there's that 😅
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The new default uses cached index and filter blocks, 16 KiB block tables, tuned Bloom filters, and bottommost ZSTD compression for SMT data. Metadata stays uncompressed. The RocksDB config also keeps support for tuning overrides, memory-budget controls, and durability mode selection.
The main issue this helps is #1776. It should also help the write-heavy performance work in #1537. In earlier quick Criterion runs, the tuned block-table profile improved vanilla insertion/100 by 14.13%, with_history insertion/10 by 9.44%, and with_history insertion/500 by 12.62%. Bottommost ZSTD improved vanilla insertion/1 by 12.39%. Read-path changes in those runs were mostly within noise, so this is best treated as a better default write profile.
This PR does not try to fix the RocksDB correctness issues in #1728, #1729, or #1730. It changes defaults and config control, not error handling or storage semantics.