Skip to content

Fee estimation: avoid serving stale fee estimate#27622

Merged
glozow merged 4 commits intobitcoin:masterfrom
ismaelsadeeq:05-2023-avoid-serving-stale-fee-estimate
Jun 20, 2023
Merged

Fee estimation: avoid serving stale fee estimate#27622
glozow merged 4 commits intobitcoin:masterfrom
ismaelsadeeq:05-2023-avoid-serving-stale-fee-estimate

Conversation

@ismaelsadeeq
Copy link
Member

@ismaelsadeeq ismaelsadeeq commented May 10, 2023

Fixes #27555

The issue arises when an old fee_estimates.dat file is sometimes read during initialization.
Or after an unclean shutdown, the latest fee estimates are not flushed to fee_estimates.dat.
If the fee estimates in the old file are old, they can cause transactions to become stuck in the mempool.
This PR ensures that nodes do not use stale estimates from the old file during initialization. If fee_estimates.dat
has not been updated for 60 hours or more, it is considered stale and will not be read during initialization. To avoid
having old estimates, the fee_estimates.dat file will be flushed periodically every hour. As mentioned #27555

"The immediate improvement would be to store fee estimates to disk once an hour or so to reduce the chance of having an old file. From there, this case could probably be detected, and refuse to serve estimates until we sync."

In addition, I will follow-up PR to persist the mempoolminfee across restarts.

@DrahtBot
Copy link
Contributor

DrahtBot commented May 10, 2023

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK willcl-ark, instagibbs, glozow
Concept ACK RandyMcMillan
Stale ACK pinheadmz

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #25380 (Detect and ignore transactions that were CPFP'd in the fee estimator by darosior)
  • #10443 (Add fee_est tool for debugging fee estimation code by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@fanquake
Copy link
Member

cc @instagibbs @TheBlueMatt

Copy link
Member

@maflcko maflcko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. Left some nits

@ismaelsadeeq ismaelsadeeq force-pushed the 05-2023-avoid-serving-stale-fee-estimate branch from a2ad5e4 to 0d2329a Compare May 11, 2023 18:43
@ismaelsadeeq ismaelsadeeq force-pushed the 05-2023-avoid-serving-stale-fee-estimate branch 2 times, most recently from 0d2329a to d24f874 Compare May 11, 2023 23:09
@ismaelsadeeq ismaelsadeeq force-pushed the 05-2023-avoid-serving-stale-fee-estimate branch from aecd92e to 3a9fee2 Compare May 18, 2023 11:01
@ismaelsadeeq ismaelsadeeq force-pushed the 05-2023-avoid-serving-stale-fee-estimate branch from 3a9fee2 to b02e3fc Compare May 18, 2023 13:14
Copy link
Member

@pinheadmz pinheadmz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

concept ACK

This should solve the stale fee issue, the test is clever and effective. I ensured the test fails without the patch and also checked the test failed when the mtime was set < 12 hours. Reviewed code and left some notes / questions

@ismaelsadeeq ismaelsadeeq force-pushed the 05-2023-avoid-serving-stale-fee-estimate branch 2 times, most recently from ef2ee18 to 473946b Compare May 18, 2023 23:02
Copy link
Member

@glozow glozow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approach ACK

@ismaelsadeeq ismaelsadeeq force-pushed the 05-2023-avoid-serving-stale-fee-estimate branch from 473946b to 1315c9f Compare May 19, 2023 16:49
@ismaelsadeeq ismaelsadeeq force-pushed the 05-2023-avoid-serving-stale-fee-estimate branch 3 times, most recently from 5f79a92 to cd9828f Compare May 20, 2023 08:54
@ismaelsadeeq ismaelsadeeq force-pushed the 05-2023-avoid-serving-stale-fee-estimate branch 2 times, most recently from de80c10 to 948edc9 Compare May 26, 2023 13:44
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jun 21, 2023
d2b39e0 test: ensure old fee_estimate.dat not read on restart and flushed (ismaelsadeeq)
cf219f2 tx fees, policy: read stale fee estimates with a regtest-only option (ismaelsadeeq)
3eb241a tx fees, policy: do not read estimates of old fee_estimates.dat (ismaelsadeeq)
5b886f2 tx fees, policy: periodically flush fee estimates to fee_estimates.dat (ismaelsadeeq)

Pull request description:

  Fixes bitcoin#27555

  The issue arises when an old `fee_estimates.dat` file is sometimes read during initialization.
  Or after an unclean shutdown, the latest fee estimates are not flushed to `fee_estimates.dat`.
  If the fee estimates in the old file are old, they can cause transactions to become stuck in the mempool.
  This  PR ensures that nodes do not use stale estimates from the old file during initialization. If  `fee_estimates.dat`
  has not been updated for 60 hours or more, it is considered stale and will not be read during initialization. To avoid
  having old estimates, the `fee_estimates.dat` file will be flushed periodically every hour. As mentioned bitcoin#27555

  > "The immediate improvement would be to store fee estimates to disk once an hour or so to reduce the chance of having an old file. From there, this case could probably be detected, and refuse to serve estimates until we sync."

  In addition, I will follow-up PR to persist the `mempoolminfee` across restarts.

ACKs for top commit:
  willcl-ark:
    ACK d2b39e0
  instagibbs:
    reACK bitcoin@d2b39e0
  glozow:
    ACK d2b39e0. One nit if you follow up.

Tree-SHA512: 4f6e0c296995d0eea5cf80c6aefdd79b7295a6a0ba446f2166f32afc105fe4f831cfda1ad3abd13c5c752b4fbea982cf4b97eaeda2af1fd7184670d41edcfeec
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request Aug 16, 2023
This reduces chances of having old estimates in fee_estimates.dat.

Github-Pull: bitcoin#27622
Rebased-From: 5b886f2
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request Aug 16, 2023
Old fee estimates could cause transactions to become stuck in the
mempool. This commit prevents the node from using stale estimates
from an old file.

Github-Pull: bitcoin#27622
Rebased-From: 3eb241a
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request Aug 16, 2023
If -acceptstalefeeestimates option is passed stale fee estimates can now
be read when operating in regtest environments.

Additionally, this commit updates all declarations of the CBlockPolicyEstimator
class to include a the second constructor variable.

Github-Pull: bitcoin#27622
Rebased-From: cf219f2
luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request Aug 16, 2023
This commit adds tests to ensure that old fee_estimates.dat files
are not read and that fee_estimates are periodically flushed to the
fee_estimates.dat file.

Additionaly it tests the -regtestonly option -acceptstalefeeestimates.

Github-Pull: bitcoin#27622
Rebased-From: d2b39e0
glozow added a commit to bitcoin-core/gui that referenced this pull request Aug 22, 2023
…` option is only supported on regtest chain

ee5a036 test: ensure acceptstalefeeestimates is supported only on regtest chain (ismaelsadeeq)
22d5d4b tx fees, policy: doc: update and delete unnecessary comment (ismaelsadeeq)

Pull request description:

  This PR Follow up comments from [#27622](bitcoin/bitcoin#27622)

  It test that the new `regtest-only` option `acceptstalefeeestimates` is not supported on [main, signet and test chains](https://github.com/bitcoin/bitcoin/pull/27622/files#r1235218268), removes an unnecessary [comment](https://github.com/bitcoin/bitcoin/pull/27622/files#r1235204323), and update fee estimator  `MAXFILEAGE` [description comment](https://github.com/bitcoin/bitcoin/pull/27622/files#r1233887314).

ACKs for top commit:
  jonatack:
    ACK ee5a036
  glozow:
    utACK ee5a036

Tree-SHA512: 4755f25b08db62f37614ea768272b12580ee0d481fb7fa339379901a6132c66828777c6747d3fe67490ceace3a6ff248bf13bdf65720f6e5ba8642eb762acd3c
Frank-GER pushed a commit to syscoin/syscoin that referenced this pull request Sep 8, 2023
… is only supported on regtest chain

ee5a036 test: ensure acceptstalefeeestimates is supported only on regtest chain (ismaelsadeeq)
22d5d4b tx fees, policy: doc: update and delete unnecessary comment (ismaelsadeeq)

Pull request description:

  This PR Follow up comments from [bitcoin#27622](bitcoin#27622)

  It test that the new `regtest-only` option `acceptstalefeeestimates` is not supported on [main, signet and test chains](https://github.com/bitcoin/bitcoin/pull/27622/files#r1235218268), removes an unnecessary [comment](https://github.com/bitcoin/bitcoin/pull/27622/files#r1235204323), and update fee estimator  `MAXFILEAGE` [description comment](https://github.com/bitcoin/bitcoin/pull/27622/files#r1233887314).

ACKs for top commit:
  jonatack:
    ACK ee5a036
  glozow:
    utACK ee5a036

Tree-SHA512: 4755f25b08db62f37614ea768272b12580ee0d481fb7fa339379901a6132c66828777c6747d3fe67490ceace3a6ff248bf13bdf65720f6e5ba8642eb762acd3c
@fanquake
Copy link
Member

Added to #28487 for backporting to 25.1.

@fanquake
Copy link
Member

Pushed into #28535 for backporting to 24.x as well.

fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Sep 26, 2023
This reduces chances of having old estimates in fee_estimates.dat.

Github-Pull: bitcoin#27622
Rebased-From: 5b886f2
fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Sep 26, 2023
Old fee estimates could cause transactions to become stuck in the
mempool. This commit prevents the node from using stale estimates
from an old file.

Github-Pull: bitcoin#27622
Rebased-From: 3eb241a
fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Sep 26, 2023
If -acceptstalefeeestimates option is passed stale fee estimates can now
be read when operating in regtest environments.

Additionally, this commit updates all declarations of the CBlockPolicyEstimator
class to include a the second constructor variable.

Github-Pull: bitcoin#27622
Rebased-From: cf219f2
fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Sep 26, 2023
This commit adds tests to ensure that old fee_estimates.dat files
are not read and that fee_estimates are periodically flushed to the
fee_estimates.dat file.

Additionaly it tests the -regtestonly option -acceptstalefeeestimates.

Github-Pull: bitcoin#27622
Rebased-From: d2b39e0
fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Oct 2, 2023
This reduces chances of having old estimates in fee_estimates.dat.

Github-Pull: bitcoin#27622
Rebased-From: 5b886f2
fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Oct 2, 2023
Old fee estimates could cause transactions to become stuck in the
mempool. This commit prevents the node from using stale estimates
from an old file.

Github-Pull: bitcoin#27622
Rebased-From: 3eb241a
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid serving stale fees