Skip to content

[EXPERIMENTAL] Schnorr batch verification for blocks#29491

Draft
fjahr wants to merge 10 commits intobitcoin:masterfrom
fjahr:2024-02-batch-validation-updated
Draft

[EXPERIMENTAL] Schnorr batch verification for blocks#29491
fjahr wants to merge 10 commits intobitcoin:masterfrom
fjahr:2024-02-batch-validation-updated

Conversation

@fjahr
Copy link
Contributor

@fjahr fjahr commented Feb 27, 2024

This is a draft implementation of Schnorr signature batch verification in blocks, put here for conceptual discussion, and collaborative benchmarking.

The secp256k1 code is still under review itself, please spend some time giving feedback there if you can:

TODOs

  • Batch taproot tweak verification as well

@DrahtBot
Copy link
Contributor

DrahtBot commented Feb 27, 2024

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/29491.

Reviews

See the guideline for information on the review process.
A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #34184 (mining: add cooldown to createNewBlock() immediately after IBD by Sjors)
  • #32575 (refactor: Remove special treatment for single threaded script checking by fjahr)
  • #32317 (kernel: Separate UTXO set access from validation functions by sedited)
  • #29843 (policy: Allow non-standard scripts with -acceptnonstdtxn=1 (test nets only) by ajtowns)
  • #28690 (build: Introduce internal kernel library by sedited)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

  • memeory -> memory [spelling error making the word unclear]

Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):

  • secp256k1_batch_add_schnorrsig(..., 32, ...) in src/batchverify.cpp

2026-01-31 14:01:32

@fjahr fjahr force-pushed the 2024-02-batch-validation-updated branch from 454783c to 0a5e612 Compare February 27, 2024 17:10
@DrahtBot
Copy link
Contributor

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

Debug: https://github.com/bitcoin/bitcoin/runs/22041704016

@Sjors
Copy link
Member

Sjors commented Feb 28, 2024

I guess this is most interesting to test against the last 50K blocks on mainnet, i.e. since early 2023? Given the high percentage of taproot transactions. Such testing could benefit from a assume utxo snapshot, saving everyone the pain of rewinding. I can bake one, though ideally I'd like #26045 to land first.

Is the current PR already using batches? It's unclear to me by glancing over validation.cpp if and how it's collecting up to max_batch_size{106} signatures before calling batch.Verify().

Maybe add a -schnorrbatchverify startup argument so more people can try it once it gets through review.

@fjahr
Copy link
Contributor Author

fjahr commented Feb 28, 2024

I guess this is most interesting to test against the last 50K blocks on mainnet, i.e. since early 2023? Given the high percentage of taproot transactions. Such testing could benefit from a assume utxo snapshot, saving everyone the pain of rewinding. I can bake one, though ideally I'd like #26045 to land first.

Yeah, that would be great. I have to get the tracing part to work in my setup again before I can start.

Is the current PR already using batches? It's unclear to me by glancing over validation.cpp if and how it's collecting up to max_batch_size{106} signatures before calling batch.Verify().

Yes, the rough architecture is as as follows (sorry for not writing a better explanation above but it's still very rough and I expect it to change). So, the batch object is constructed and then handed down from ConnectBlock() to CheckInputScripts() which hands it to the CScriptCheck constructor. When the CScriptCheck instance is executed and the batch object is present BatchingCachingTransactionSignatureChecker is used which only differs from the default CachingTransactionSignatureChecker in that its implementation of VerifySchnorrSignature() and adds the Schnorr sig to the batch object instead of verifying it right there. (Here we have an issue because the function is not doing what the name says anymore but it is a much simpler change this way and I find it hard to predict where we will land with this in the end.) Then back in ConnectBlock() the batch object is verified after all transactions have been iterated over.

The 106 is an implementation detail from secp256k1 that I am not sure needs to be really exposed here. But what matters for the question is: if there are more than 106 sigs added to the batch the verification for the already added sigs will happen when the next sig is added and if the verification fails the adding itself will fail. So, a callback to the last paragraph, every 106 sigs the naming of VerifySchnorrSignature() is actually still correct ;) and the one verify call in the end just verifies the last n % 106 sigs left for that block. I did not put any effort in documenting this properly here yet because the secp256k1 API is not finalized yet and the details on this particular topic might still change, see for example bitcoin-core/secp256k1#1134 (review)

Everything should be conditional on the presence of a batch object which defaults to nullptr and if that is the case all the other functions should work as usual, for example when used outside of the context of a new block.

Maybe add a -schnorrbatchverify startup argument so more people can try it once it gets through review.

Yeah, I have been thinking that or maybe even a compiler argument

@fjahr fjahr force-pushed the 2024-02-batch-validation-updated branch from 0a5e612 to 1825555 Compare February 28, 2024 13:14
@fjahr
Copy link
Contributor Author

fjahr commented Feb 28, 2024

@Sjors maybe what confused you is that BatchingCachingTransactionSignatureChecker::VerifySchnorrSignatureBatch() was unused. That was actually a leftover from an earlier design spike and I just forgot to remove it. It's gone now.

@Sjors
Copy link
Member

Sjors commented Feb 28, 2024

if there are more than 106 sigs added to the batch the verification for the already added sigs will happen when the next sig is added and if the verification fails the adding itself will fail.

That clarifies a lot, and should be documented :-)

@0xB10C
Copy link
Contributor

0xB10C commented May 13, 2024

@willcl-ark and I ran a IBD benchmark vs master for this PR: IBD from a local node to height 840000 and currently only looking at the connect_block duration.

In the current draft implementation:

  • master is currently faster
  • batch validation is attempted and decreases performance even before taproot activated
  • after taproot activation, batch validation is significantly slower than non-batch validation

image

image

Used this bpftrace script to collect CSV data:

#!/usr/bin/env bpftrace

usdt:./src/bitcoind:validation:block_connected
{
  $height = (int32) arg1;
  $tx = (uint64) arg2;
  $ins = (int32) arg3;
  $sigops = (int64) arg4;
  $duration = (int64) arg5; // in nanoseconds

  printf("%d,%ld,%d,%ld,%ld\n", $height, $tx, $ins, $sigops, $duration);
}

@willcl-ark
Copy link
Member

batch validation is attempted and decreases performance even before taproot activated

From a quick skim of the changes ISTM that we always use the BatchingCachingTransactionSignatureChecker, and there was no switch to the CachingTransactionSignatureChecker, but this could well be because this is only in WIP state. It doesnt account for why it's currently slower in all cases though.

@fjahr fjahr force-pushed the 2024-02-batch-validation-updated branch from 1825555 to 6f271d7 Compare December 5, 2024 13:57
@DrahtBot DrahtBot mentioned this pull request Dec 6, 2024
@andrewtoth
Copy link
Contributor

It seems in this approach the m_batch_mutex is held for the entirety of adding, which will cause lock contention. Verify is then a blocking call on the main thread, which negates the multithreading.

I think a better approach would be to have a thread local batch for each worker thread and the main thread, so each thread can add all schnorr sigs without locking, and then each thread can verify their batches in parallel at the end.

@fjahr fjahr force-pushed the 2024-02-batch-validation-updated branch from 6f271d7 to cafdd14 Compare March 11, 2025 23:43
@fjahr fjahr changed the title [DO NOT MERGE] Schnorr batch verification for blocks [EXPERIMENTAL] Schnorr batch verification for blocks Mar 11, 2025
@fjahr fjahr force-pushed the 2024-02-batch-validation-updated branch from bcbbb57 to 452a375 Compare January 26, 2026 15:56
Copy link
Contributor Author

@fjahr fjahr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased on latest master and included the latest API changes in the batch module PR in secp256k1.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not discarding it anymore in the latest push.

@DrahtBot
Copy link
Contributor

🚧 At least one of the CI tasks failed.
Task fuzzer,address,undefined,integer: https://github.com/bitcoin/bitcoin/actions/runs/21364278324/job/61491586498
LLM reason (✨ experimental): Fuzz target crash: libFuzzer reported a deadly signal (exit code 77) during fuzz run.

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

fjahr and others added 10 commits January 31, 2026 14:28
15ea24cb8c batch: make add functions void & introduce reset
bfcc479a35 batch: remove `batch_usable` api
15e388e096 batch: make tests functions internal & static
aac054a373 fix typos & index the right inputs for benchmarks
c07e710003 batch: remove experimental status
49fb753393 test: fix ci failures
e96dabb4af batch: Generate speedup graphs
b0b3425cd4 batch, extrakeys: Add benchmarks
9d5115156b batch: Add tests for batch_add_* APIs
668199c917 batch,ecmult: Add tests for core batch APIs and strauss_batch refactor
53a158203f batch: Add example
b40b4186b8 batch: Add batch_add_* APIs
2bed1cb6ee batch, ecmult: Add batch_verify and refactor strauss_batch
8f13eeae31 batch: Add create and destroy APIs
0b6b0c87ad batch: Initialize an experimental batch module
REVERT: 14e56970cb Merge bitcoin-core/secp256k1#1794: ecmult: Use size_t for array indices
REVERT: c7a52400d6 Merge bitcoin-core/secp256k1#1809: release cleanup: bump version after 0.7.1
REVERT: ae7eb729c0 release cleanup: bump version after 0.7.1
REVERT: 1a53f4961f Merge bitcoin-core/secp256k1#1808: Prepare for 0.7.1
REVERT: 20a209f11c release: prepare for 0.7.1
REVERT: c4b6a81a60 changelog: update in preparation for the v0.7.1 release
REVERT: ebb35882da Merge bitcoin-core/secp256k1#1796: bench: fail early if user inputs invalid value for SECP256K1_BENCH_ITERS
REVERT: c09215f7af bench: fail early if user inputs invalid value for SECP256K1_BENCH_ITERS
REVERT: 471e3a130d Merge bitcoin-core/secp256k1#1800: sage: verify Eisenstein integer connection for GLV constants
REVERT: 29ac4d8491 sage: verify Eisenstein integer connection for GLV constants
REVERT: 4721e077b4 Merge bitcoin-core/secp256k1#1793: doc/bench: added help text for SECP256K1_BENCH_ITERS env var for bench_ecmult
REVERT: bd5ced1fe1 doc/bench: added help text for SECP256K1_BENCH_ITERS env var for bench_ecmult
REVERT: 47eb70959a ecmult: Use size_t for array indices in _odd_multiplies_table
REVERT: bb1d199de5 ecmult: Use size_t for array indices into tables
REVERT: 2d9137ce9d Merge bitcoin-core/secp256k1#1764: group: Avoid using infinity field directly in other modules
REVERT: f9a944ff2d Merge bitcoin-core/secp256k1#1790: doc: include arg -DSECP256K1_USE_EXTERNAL_DEFAULT_CALLBACKS=ON for cmake
REVERT: 0406cfc4d1 doc: include arg -DUSE_EXTERNAL_DEFAULT_CALLBACKS=1 for cmake
REVERT: 8d445730ec Merge bitcoin-core/secp256k1#1783: Add VERIFY_CHECKs and documentation that flags must be 0 or 1
REVERT: aa2a39c1a7 Merge bitcoin-core/secp256k1#1778: doc/bench: Added cmake build options to bench error messages
REVERT: 540fec8ae9 Merge bitcoin-core/secp256k1#1788: test: split monolithic ellswift test into independent cases
REVERT: d822b29021 test: split monolithic ellswift test into independent cases
REVERT: ae00c552df Add VERIFY_CHECKs that flags are 0 or 1
REVERT: 5c75183344 Merge bitcoin-core/secp256k1#1784: refactor: remove ret from secp256k1_ec_pubkey_serialize
REVERT: be5e4f02fd Merge bitcoin-core/secp256k1#1779: Add ARG_CHECKs to ensure "array of pointers" elements are non-NULL
REVERT: 3daab83a60 refactor: remove ret from secp256k1_ec_pubkey_serialize
REVERT: 8bcda186d2 test: Add non-NULL checks for "pointer of array" API functions
REVERT: 5a08c1bcdc Add ARG_CHECKs to ensure "array of pointers" elements are non-NULL
REVERT: 3b5b03f301 doc/bench: Added cmake build options to bench error messages
REVERT: e7f7083b53 Merge bitcoin-core/secp256k1#1774: refactor: split up internal pubkey serialization function into compressed/uncompressed variants
REVERT: b6c2a3cd77 Merge bitcoin-core/secp256k1#1761: ecmult_multi: reduce strauss memory usage by 30%
REVERT: f5e815f430 remove secp256k1_eckey_pubkey_serialize function
REVERT: 0d3659c547 use new `_eckey_pubkey_serialize{33,65}` functions in modules (ellswift,musig)
REVERT: adb76f82ea use new `_eckey_pubkey_serialize{33,65}` functions in public API
REVERT: fc7458ca3e introduce `secp256k1_eckey_pubkey_serialize{33,65}` functions
REVERT: 2f73e5281d group: Avoid using infinity field directly in other modules
REVERT: 26166c4f5f ecmult_multi: reduce strauss memory usage by 30%

git-subtree-dir: src/secp256k1
git-subtree-split: 15ea24cb8c1bd239a7a39939da1952cf6d3a35b0
@fjahr fjahr force-pushed the 2024-02-batch-validation-updated branch from 41e0ea6 to f178c6c Compare January 31, 2026 14:00
@DrahtBot
Copy link
Contributor

DrahtBot commented Feb 3, 2026

🐙 This pull request conflicts with the target branch and needs rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants