fees: Introduce Mempool Based Fee Estimation to reduce overestimation by ismaelsadeeq · Pull Request #34075 · bitcoin/bitcoin

ismaelsadeeq · 2025-12-15T11:53:35Z

This PR is another attempt to fix #27995 using a better approach.

For background and motivation, see #27995 and the discussion in the Delving Bitcoin post Mempool Based Fee Estimation on Bitcoin Core
This PR is currently limited to using the mempool only to lower what is recommended by the Block Policy Estimator.
Accurate and safe fee estimation using the mempool is challenging. There are open questions about how to prevent mempool games that are theoretically possible for miners (a variant of the Finney attack)

This is one of the reasons I opted to use the mempool only to lower the Block Policy Estimator, which itself is not gameable, and therefore this mechanism is not susceptible to this attack.

The underlying assumption here is that, with the current tools and work done to make RBF and CPFP feasible and guarantee (TRUC transaction relay, ephemeral anchors, cluster size 2 package rbf), it is safer to underestimate rather than overestimate.
We now assume it is relatively easy to fee-bump later if a transaction does not confirm, whereas once a fee is overestimated there is no way to recover from that.

Another open question when using the mempool for fee estimation is how to account for incoming transaction inflow.
Bitcoin Augur does this by using past inflow plus a constant expected inflow to predict future inflow. I find this unconvincing for fee estimation and potentially prone to more overestimation, as past conditions are not always representative of the future. See my review of the Augur fee rate estimator and open questions.

This PR uses a much simpler approach based on current user behavior, similar to the widely used method employed by mempool.space: looking at the top block of the mempool and selecting a percentile feerate depending on whether the user is economical or conservative.

Empirical data from both myself and Clara Shikhelman shows that the 75th percentile feerate for economical users and the 50th percentile feerate for conservative users provide positive confirmation guarantees, hence this is what is used in this PR.

Parallel research by René Pickhardt and his student suggests that using the average fee per byte of the block template performs well.

All of these are constants that can be adjusted, there is parallel work exploring these constants and running benchmarks across fee estimators to find a sweet spot.

See also work in LND, the LND Budget Sweeper, which applies this idea successfully. Their approach is to estimate fees initially with bitcoind, then increment gradually as the confirmation deadline approaches, using a fixed fee budget.

Historical data indicates that by doing only the approach in this PR, we are able to reduce overestimation quite significantly (~29%).

This is particularly useful in scenarios where the Block Policy Estimator recommends a high feerate while the mempool is empty.

As seen in the image above, there is only one remaining unfixed case: when there is a sudden inflow of transactions and the feerate rises, the Block Policy Estimator takes time to reflect this. In that case, users will continue to see a low feerate estimate until it slowly updates. From the historical data linked above, this occurs about ~26% of the time).

Overall, we observe a 73% success rate with 0% overestimation, and 26% underestimation with this approach.

See https://bitcoincorefeerate.com/stats for recent running stats that have almost identical data.

This PR also includes refactors that enable this work. Rather than splitting the PR and implementing changes incrementally, I opted for an end-to-end implementation:

1. Refactors

Separation of feerate format from the fee estimate mode enumerator (separation of concerns, as they are semantically distinct)
Move logging after fee estimation estimateSmartFee from the wallet to the Block Policy Estimator internal method (also separation of concerns and avoiding leaking fee estimator internals). This change also comes with updating createTransactionInternal to log only the fee and its source (fee estimator, fallback fee, etc.), which is more appropriate.
Separate the fee estimate mode enum from the fee source enumerator (separation of concerns, as they are semantically distinct). This introduces a minor breaking change: the wallet RPCs now return fee source instead of fee reason, which is more accurate. Let me know if we prefer to keep the field name unchanged while returning the fee source.
Move the StringForReason enum from common to the Block Policy Estimator (also separation of concerns)
Various test file updates and renames for improved accuracy
Update Block Policy Estimator unit tests to be independent of the mempool and the validation interface

2. Introduce Mempool-Based Fee Estimator and Fee Estimator Manager

Introduce a new abstract fee estimator base class that all fee estimators must implement
Introduce FeeRateEstimateResult as the response type, containing metadata and avoiding the use of out-parameters. This is easily extensible and makes future changes less invasive, as method signatures do not need to change.
Add FeeEstimatorType enum to identify different fee estimators
Make CBlockPolicyEstimator derive from the new fee estimator base class
Add FeeRateEstimatorManager, responsible for managing all fee estimators
Update the node context to store a unique_ptr to FeeRateEstimatorManager instead of CBlockPolicyEstimator
Update CBlockPolicyEstimator to no longer subscribe directly to the validation interface; instead, FeeRateEstimatorManager subscribes and forwards relevant notifications
Add a percentile calculation function that takes the chunk feerates of a block template and returns feerate percentiles; this can also be reused by the getblockstats RPC
Add a mempool fee estimator that generates a block template when called, calculates a percentile feerate, and returns the 75th percentile for economical mode or the 50th percentile for conservative mode
Add caching to the mempool estimator so that new estimates are generated only every 7 seconds, assuming enough transactions have propagated to make a meaningful difference.
This heuristic will likely be replaced by requesting block templates via the general-purpose block template cache proposed here: RFC: Bitcoin Core Node BlockTemplateManager #33389
Update MempoolTransactionRemovedForBlock to return the actual block transactions as well
Track the weight of block transactions and evicted transactions after each block connection
This data is tracked for the last 6 mined blocks. A mempool feerate estimate is returned only when the average ratio of mempool transaction weight evicted due to BLOCK reasons to block transaction weight is greater than 75%. This heuristic provides rough confidence that the node’s mempool matches that of the majority of the hashrate. The 75% threshold is arbitrary and can be adjusted.

There is a caveat when transactions in the local mempool are consistently not mined by the network, as described in #27995 (e.g., due to filtering).
Accounting for these transactions during fee estimation is not necessary, as they should be evicted from the mempool itself (see #33510). Handling this again within fee estimation would be redundant.

Persist statistics for the 6 most recent blocks to disk during periodic flushes and shutdown, so this data is available after restarts
Expose this data to users when running estimatesmartfee with verbosity > 2

see example output

bitcoin-cli estimatesmartfee 1 "conservative" 2

{
  "feerate": 0.00002133,
  "errors": [],
  "blocks": 2,
  "mempool_health_statistics": [
    {
      "block_height": 927953,
      "block_weight": 3991729,
      "mempool_txs_weight": 3942409,
      "ratio": 0.987644451815241
    },
    ...
  ]
}

DrahtBot · 2025-12-15T11:53:44Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/34075.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	sedited

If your review is incorrectly listed, please copy-paste  into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#34806 (refactor: logging: Various API improvements by ajtowns)
#34803 (mempool: asynchronous mempool fee rate diagram updates via validation interface by ismaelsadeeq)
#34704 (validation: Explicitly move blocks to validation signals by sedited)
#34684 (refactor: Enable -Wswitch in exhaustive switch'es, Enable -Wcovered-switch-default by maflcko)
#34617 (fees: wallet: remove block policy fee estimator internals from wallet by ismaelsadeeq)
#34405 (wallet: skip APS when no partial spend exists by 8144225309)
#33922 (mining: add getMemoryLoad() and track template non-mempool memory footprint by Sjors)
#33637 (refactor: optimize block index comparisons (1.4-6.8x faster) by l0rinc)
#33421 (node: add BlockTemplateCache by ismaelsadeeq)
#32554 (bench: replace embedded raw block with configurable block generator by l0rinc)
#32427 ((RFC) kernel: Replace leveldb-based BlockTreeDB with flat-file based store by sedited)
#28690 (build: Introduce internal kernel library by sedited)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

!Checks -> Checks [Leading exclamation mark makes the sentence look like a typo; remove the "!" so the comment reads normally.]

Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):

node0.estimatesmartfee(1, "economical", 2) in test/functional/feature_fee_estimation.py
assert_raises_rpc_error(-1, "estimatesmartfee", self.nodes[0].estimatesmartfee, 1, 'ECONOMICAL', 1, 1) in test/functional/rpc_estimatefee.py

^{2026-02-26 16:47:35}

gmaxwell · 2025-12-15T23:01:56Z

I think this is a good idea, especially if it there was a 'very conservative' that just didn't use this.

Accounting for these transactions during fee estimation is not necessary, as they should be evicted from the mempool itself

I don't agree here-- rather time based expiration should probably be eliminated over time. If there really is a censored transaction then we want nodes to work on getting it mined, and not just give up if the censors manage to control the blocks for a long time.

Also two weeks is already long time to allow a non-minable transaction to distort fee estimates. And it's a short time for low rate confirmation-- there have been times in history when transactions weren't confirmed for months after their initial announcement (though those were also at time so far where size based expiration would have eliminated them).

So I suggest instead that future work (not this PR) could improve the estimator by excluding from considerations transactions that should have been mined according to our template but weren't for more than a threshold number of blocks, perhaps 6 or something like that.

Such code could also be used to log/warn the user that there is either ongoing censorship or that their node's policy appears to be less restrictive than the network.

But in any case it doesn't need to be done immediate to make use of this kind of estimator.

Overall, we observe a 73% success rate with 0% overestimation, and 26% underestimation with this approach.

One thing to keep in mind is that if this approach is widely used the success rate will probably change. It might make sense to initially deploy with more conservative settings (/defaults) and slowly increase them to avoid causing an abrupt change.

ismaelsadeeq · 2025-12-16T15:39:16Z

I don't agree here; rather, time-based expiration should probably be eliminated over time. If there really is a censored transaction, then we want nodes to continue working to get it mined, not just give up if censors manage to control blocks for a long time.

Indeed. In the context of censorship, we would want to keep such transactions if they are incentive compatible. However, there may also be legitimate reasons miners deliberately exclude them for example, a legacy node that is unaware of a soft fork that invalidates a certain spend (e.g., post-quantum).

Hence as you also mentioned, I would like to see the direction of #33510, leaning toward deferring this improvement to future work.

Especially if there were a “very conservative” option that simply didn’t use this. One thing to keep in mind is that if this approach is widely used, the success rate will probably change. It might make sense to initially deploy with more conservative settings (or defaults) and slowly increase them to avoid causing an abrupt change.

I considered this and even had a branch that added an option to estimatesmartfee to force block-policy-only behavior.

However, the users this targets are those running bitcoind with default settings and trusting developers to make sane default choices. I’ve seen this scenario before with previous conservative defaults: complaints and issues accumulated over time, eventually leading us to switch block policy the default to economical, which provided a significant reduction in overestimation and better accuracy.

FWIW, I also did some research over a short block interval to get a sense of what Bitcoin core wallet users uses for feerate estimation, and I find that the majority of Bitcoin Core wallet users (that I was able to fingerprints) are not using the block-policy estimator, we’ve seen self-hosted users 1 2 switch approaches in the past, I’ve also personally spoken to large Bitcoin services that rely on third party but want this feature.
It seems to me enabling this by default feels like the better direction, IMHO.

gmaxwell · 2025-12-17T14:57:03Z

I think enabling it by default probably makes sense, that particular sub comment was more about having a ready way to choose not to use it when transacting.

As far as the initial defaults I think there I meant that that it shouldn't target too close to the bottom of the mempool as even if that currently gets high success that may not be true after wide deployment. Though on reflection I'm not sure that thats true as this only lowers feerates, so other people using this should only increase your odds of success while using it.

- Initialize FeeEstimatorManager in node instead of BlockPolicyEstimator. - Forward fee rate estimation requests through FeeEstimatorManager. - Update Chain interface to return FeeRateEstimatorResult, removing the out parameter requirement. - Retain selected BlockPolicyEstimator methods for test-only estimaterawfee RPC compatibility.

- Given a vector of chunk feerates in the order they were added to the block, CalculatePercentiles function will return the 25th, 50th, 75th and 95th percentile feerates. - This commit also add a unit test for the CalculatePercentile function.

- This will prevents "node/context -> policy/fees/estimator_man -> policy/fees/mempool_estimator -> node/miner -> node/context" circular dependency

- The mempool fee rate estimator uses the unconfirmed transactions in the mempool to generate a fee rate that a package should have for it to confirm as soon as possible. Co-authored-by: willcl-ark <[email protected]>

- Only refresh the cached fee rate estimate if it is older than 7 seconds. - This avoids generating block templates too often. Co-authored-by: willcl-ark <[email protected]>

…s or not

…lock`

- This ensures that the users mempool has seen at least 75% of the last 6 blocks transactions.

…s.dat

- This commit also enable estimatesmartfee to return the stats when verbosity level is greater than 1.

ismaelsadeeq · 2026-03-05T11:50:59Z

See bitcoincorefeerate.com/stat for a recent running benchmark that measures the accuracy of the fee rate estimates being returned by this PR

sedited · 2026-03-14T08:18:21Z

Concept ACK

People have been discussing and asking for this for a long time already, cool to see this actually implemented!

for a recent running benchmark that measures the accuracy of the fee rate estimates being returned by this PR

Do you already have any data on how this accuracy compares to some of the external services, like bitcoin augur or mempool.space?

ismaelsadeeq · 2026-03-16T07:31:42Z

Do you already have any data on how this accuracy compares to some of the external services, like bitcoin augur or mempool.space?

Previous benchmarks were done relative to our current fee rate estimator (Block Policy). This work aims to improve Block Policy fee rate estimator by reducing overestimation when the mempool is empty.

mempool.space is fully mempool-based and can be more accurate than this during high mempool activity. We choose to be conservative during high mempool activity and use a block policy estimator to avoid being susceptible to mempool games.

Block policy estimator has another advantage that mempool.space does not have, which is fee rate estimates up to 1008 blocks with a high probability of success within that range, for people who want to pay lower fees at the cost of waiting, and this is actively used by lightning implementations e.g LND budget sweeper.

Bitcoin Augur was benchmarked by the Block engineers, but against a stale version of Core v27.2, and significant changes were made to our fee estimator after that version, so it is not a good representation. Their result is:
https://delvingbitcoin.org/t/augur-block-s-open-source-bitcoin-fee-estimation-library/1848

Provider	Miss Rate	Avg Overestimate	Total Difference
Augur	14.1%	15.9%	13.6%
Blockstream (Core v27.2)	18.7%	44.2%	35.9%

The data and tool were open source, and I reproduced these numbers locally (Assuming the data points are correct :))
Overestimation is still a notable concern here.

I think for now, this is a good direction to go. This PR made a series of refactors and introduced a fee estimation manager and fee estimator abstract class that should ideally make it easy to add other strategies when they are studied well enough and have rough conceptual agreement.

Running benchmarks for all available fee estimators can be beneficial. I have an issue on bitcoin-core-fees

DrahtBot added the TX fees and policy label Dec 15, 2025

This was referenced Dec 15, 2025

Fees: add Fee rate Forecaster Manager #31664

Closed

Fee Estimation via Fee rate Forecasters #30157

Closed

Fee Estimation via Fee rate Forecasters tracking issue #30392

Closed

DrahtBot added the CI failed label Dec 15, 2025

ismaelsadeeq force-pushed the 12-2025-fee-estimation-improvement branch from dafb945 to 6ab7665 Compare December 15, 2025 13:47

DrahtBot removed the CI failed label Dec 15, 2025

DrahtBot added the Needs rebase label Dec 16, 2025

ismaelsadeeq force-pushed the 12-2025-fee-estimation-improvement branch from 6ab7665 to 41076c6 Compare December 16, 2025 15:36

This was referenced Dec 16, 2025

(RFC) kernel: Replace leveldb-based BlockTreeDB with flat-file based store #32427

Open

wallet, rpc: remove settxfee and paytxfee #32138

Merged

DrahtBot removed the Needs rebase label Dec 16, 2025

DrahtBot added the Needs rebase label Dec 18, 2025

ismaelsadeeq force-pushed the 12-2025-fee-estimation-improvement branch from 41076c6 to fee2612 Compare December 22, 2025 13:16

DrahtBot removed the Needs rebase label Dec 22, 2025

DrahtBot mentioned this pull request Dec 31, 2025

node: add BlockTemplateCache #33421

Open

ismaelsadeeq and others added 11 commits February 26, 2026 12:11

fees: add CalculatePercentiles function

e8cd040

- Given a vector of chunk feerates in the order they were added to the block, CalculatePercentiles function will return the 25th, 50th, 75th and 95th percentile feerates. - This commit also add a unit test for the CalculatePercentile function.

refactor: miner: remove unused node/context.h import

2f9e3f1

- This will prevents "node/context -> policy/fees/estimator_man -> policy/fees/mempool_estimator -> node/miner -> node/context" circular dependency

fees: add MemPoolFeeRateEstimator class

d31ef6c

- The mempool fee rate estimator uses the unconfirmed transactions in the mempool to generate a fee rate that a package should have for it to confirm as soon as possible. Co-authored-by: willcl-ark <[email protected]>

fees: add FeeRateEstimatorTypeToString method

dac14ca

test: add mempool fee estimator unit test

0a093b0

fees: add caching to MemPoolFeeRateEstimator

e589168

- Only refresh the cached fee rate estimate if it is older than 7 seconds. - This avoids generating block templates too often. Co-authored-by: willcl-ark <[email protected]>

fees: return mempool estimates when it's lower than block policy

aff0f2f

rpc:fees: update estimatesmartfee to return errors whether it succeed…

5aef64e

…s or not

test: ensure estimatesmartfee returns mempool fee rate estimate

310717e

validation: add block transactions to `MempoolTransactionsRemovedForB…

9b60cf3

…lock`

ismaelsadeeq force-pushed the 12-2025-fee-estimation-improvement branch from 422d4f7 to 97401b3 Compare February 26, 2026 13:58

DrahtBot removed the Needs rebase label Feb 26, 2026

ismaelsadeeq added 6 commits February 26, 2026 16:38

fees: only return mempool fee rate estimate when mempool is healthy

fec85f0

- This ensures that the users mempool has seen at least 75% of the last 6 blocks transactions.

fees: return mempool health stats from feerate estimator manager

6d91226

fees: rename and move fee_estimates.dat to fees/block_policy_estimate…

631e031

…s.dat

fees: add EncodedBlockDataFormatter struct

c9cc981

fees: persist mempool estimator statistics

af8d5ad

- This commit also enable estimatesmartfee to return the stats when verbosity level is greater than 1.

test: add mempool estimator i/o fuzz test

aeee300

ismaelsadeeq force-pushed the 12-2025-fee-estimation-improvement branch from 97401b3 to aeee300 Compare February 26, 2026 16:46

DrahtBot removed the CI failed label Feb 26, 2026

This was referenced Mar 14, 2026

refactor: logging: Various API improvements #34806

Open

mempool: asynchronous mempool fee rate diagram updates via validation interface #34803

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fees: Introduce Mempool Based Fee Estimation to reduce overestimation#34075

fees: Introduce Mempool Based Fee Estimation to reduce overestimation#34075
ismaelsadeeq wants to merge 31 commits intobitcoin:masterfrom
ismaelsadeeq:12-2025-fee-estimation-improvement

ismaelsadeeq commented Dec 15, 2025 •

edited

Loading

Uh oh!

DrahtBot commented Dec 15, 2025 •

edited

Loading

Uh oh!

gmaxwell commented Dec 15, 2025

Uh oh!

ismaelsadeeq commented Dec 16, 2025

Uh oh!

gmaxwell commented Dec 17, 2025

Uh oh!

ismaelsadeeq commented Mar 5, 2026

Uh oh!

sedited commented Mar 14, 2026

Uh oh!

ismaelsadeeq commented Mar 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

ismaelsadeeq commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Refactors

2. Introduce Mempool-Based Fee Estimator and Fee Estimator Manager

Uh oh!

DrahtBot commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage & Benchmarks

Reviews

Conflicts

LLM Linter (✨ experimental)

Uh oh!

gmaxwell commented Dec 15, 2025

Uh oh!

ismaelsadeeq commented Dec 16, 2025

Uh oh!

gmaxwell commented Dec 17, 2025

Uh oh!

ismaelsadeeq commented Mar 5, 2026

Uh oh!

sedited commented Mar 14, 2026

Uh oh!

ismaelsadeeq commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ismaelsadeeq commented Dec 15, 2025 •

edited

Loading

DrahtBot commented Dec 15, 2025 •

edited

Loading

ismaelsadeeq commented Mar 16, 2026 •

edited

Loading