build: Introduce internal kernel library by sedited · Pull Request #28690 · bitcoin/bitcoin

sedited · 2023-10-20T11:01:03Z

This PR introduces a new libbitcoin_kernel internal library. It completes the internal library design as laid out in doc/design/libraries.md. Since the util library contains a bunch of modules that are not required by the kernel library, a new kernel_util library is introduced as well that only consists of the modules required by the kernel library. The external libbitcoinkernel library now re-uses the compiled objects from the internal libraries.

There is a trade-off to this. Since we don't manually export symbols from the kernel library yet, this would make the library unusable if REDUCE_EXPORTS=ON. For now this means that the patch here introduces a separate source file list for the external library that gets populated by the source files of its static dependents.

DrahtBot · 2023-10-20T11:01:06Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/28690.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept ACK	stickies-v, theuni, kashifs, pablomartin4btc, ismaelsadeeq
Approach ACK	hebasto
Stale ACK	ryanofsky, yuvicc, janb84

If your review is incorrectly listed, please copy-paste  into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#34654 ([RFC] BlockMap and CChain Concurrency Improvement by alexanderwiederin)
#34641 (node: scale default -dbcache with system RAM by l0rinc)
#34495 (Replace boost signals with minimal compatible implementation by theuni)
#34075 (fees: Introduce Mempool Based Fee Estimation to reduce overestimation by ismaelsadeeq)
#32427 ((RFC) kernel: Replace leveldb-based BlockTreeDB with flat-file based store by sedited)
#32297 (bitcoin-cli: Add -ipcconnect option by ryanofsky)
#31132 (validation: fetch block inputs on parallel threads by andrewtoth)
#26022 (Add util::ResultPtr class by ryanofsky)
#25665 (refactor: Add util::Result failure types and ability to merge result values by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

src/Makefile.am

hebasto · 2023-10-21T10:46:44Z

To fix the MSVC build, suggesting to apply the diff as follows:

--- a/build_msvc/bitcoin.sln
+++ b/build_msvc/bitcoin.sln
@@ -82,6 +82,10 @@ Global
         {460FEE33-1FE1-483F-B3BF-931FF8E969A5}.Debug|x64.Build.0 = Debug|x64
         {460FEE33-1FE1-483F-B3BF-931FF8E969A5}.Release|x64.ActiveCfg = Release|x64
         {460FEE33-1FE1-483F-B3BF-931FF8E969A5}.Release|x64.Build.0 = Release|x64
+        {A938586F-1016-4C60-9B82-56123264EE14}.Debug|x64.ActiveCfg = Debug|x64
+        {A938586F-1016-4C60-9B82-56123264EE14}.Debug|x64.Build.0 = Debug|x64
+        {A938586F-1016-4C60-9B82-56123264EE14}.Release|x64.ActiveCfg = Release|x64
+        {A938586F-1016-4C60-9B82-56123264EE14}.Release|x64.Build.0 = Release|x64
         {5724BA7D-A09A-4BA8-800B-C4C1561B3D69}.Debug|x64.ActiveCfg = Debug|x64
         {5724BA7D-A09A-4BA8-800B-C4C1561B3D69}.Debug|x64.Build.0 = Debug|x64
         {5724BA7D-A09A-4BA8-800B-C4C1561B3D69}.Release|x64.ActiveCfg = Release|x64

sedited · 2023-10-21T14:31:49Z

Thanks for the fixes @hebasto! Looks like the remaining test failure is a timeout?

hebasto · 2023-10-21T15:36:19Z

Looks like the remaining test failure is a timeout?

I believe that this PR changes are unrelated to that timeout. See: #28703.

sedited · 2025-11-25T17:08:26Z

Rebased a9ec7cd -> 4c387e9 (kernelInternalLib_21 -> kernelInternalLib_22, compare)

janb84

ACK 4c387e9

Looks good to me, next to several builds & tests, I did a (subsection of a ) Guix build to see if there where any issues with that. It ran fine, see below.

my Guix Build Output

Host architecture: aarch64
Commit: 4c387e938015

0eea1fa55868cf288544de481d7ae5e8285f068074536723e927c20b9ece0518  guix-build-4c387e938015/output/dist-archive/bitcoin-4c387e938015.tar.gz
bf648f3ef27667e011b2c4547987527111c3d5c286933d092ad82697181c7918  guix-build-4c387e938015/output/x86_64-apple-darwin/SHA256SUMS.part
da7d979957fb90516e47e1c2a976888e934776eee59f3248be798ffb57f2a781  guix-build-4c387e938015/output/x86_64-apple-darwin/bitcoin-4c387e938015-x86_64-apple-darwin-codesigning.tar.gz
902f1ed15f8bae87129447d24de0356ec20781a813808e1b0274c815ac89072f  guix-build-4c387e938015/output/x86_64-apple-darwin/bitcoin-4c387e938015-x86_64-apple-darwin-unsigned.tar.gz
6c23aa0d3c94029af56e0573927f5d634cd32444c5cebf97f405864fb2e30139  guix-build-4c387e938015/output/x86_64-apple-darwin/bitcoin-4c387e938015-x86_64-apple-darwin-unsigned.zip
dfa686c73f0f54f71d5d81cba48e8d809106ca708c2619b641df412a48d43efa  guix-build-4c387e938015/output/x86_64-w64-mingw32/SHA256SUMS.part
34f7c1142fcab95318de2c15b10e3ef2f511137c3da196cf275061d2e7921095  guix-build-4c387e938015/output/x86_64-w64-mingw32/bitcoin-4c387e938015-win64-codesigning.tar.gz
9c6aec8c60d30ef857da2538c939f0d16642bb05f0dd50f2d282ea56c533f2cf  guix-build-4c387e938015/output/x86_64-w64-mingw32/bitcoin-4c387e938015-win64-debug.zip
45132cb1e70e6930ad4ca85ab9ae6258e61eeee4aef8858a4902e8dceb71db5e  guix-build-4c387e938015/output/x86_64-w64-mingw32/bitcoin-4c387e938015-win64-setup-unsigned.exe
7bffc2f307b721320cc5358ce79945fe241eec208b86d969affecbe5d14f9667  guix-build-4c387e938015/output/x86_64-w64-mingw32/bitcoin-4c387e938015-win64-unsigned.zip

sedited · 2025-12-09T12:17:01Z

Rebased 4c387e9 -> 5656bfa (kernelInternalLib_22 -> kernelInternalLib_23, compare)

janb84

re ACK 5656bfa

changes since last ACK:

rebase

maflcko · 2025-12-09T12:40:38Z

It could make sense to keep ../util/expected.cpp? Otherwise, iwyu won't run and the file could just be fully deleted?

sedited · 2025-12-09T12:51:24Z

It could make sense to keep ../util/expected.cpp? Otherwise, iwyu won't run and the file could just be fully deleted?

That was not the intention, forgot that it was just empty for now and was expecting a compile error in case I missed it. Fixed it now.

yuvicc · 2025-12-16T09:00:37Z

re-ACK 5f096da

Only rebased since last review.

janb84

re ACK 5f096da

changes since last ACK:

added expected.cpp to util for IWYU

sedited · 2026-01-13T15:31:01Z

Rebased 5656bfa -> ba91c51 (kernelInternalLib_23 -> kernelInternalLib_24, compare)

Resolved conflict with Broadcast own transactions only via short-lived Tor or I2P connections #29415

sedited · 2026-01-19T21:01:05Z

Rebased ba91c51 -> b828264 (kernelInternalLib_24 -> kernelInternalLib_25, compare)

Fixed conflict with refactor: [move-only] Merge core_io module, remove from libkernel #34296

sedited · 2026-02-20T10:23:59Z

Rebased b828264 -> db85819 (kernelInternalLib_25 -> kernelInternalLib_26, compare)

Fixed conflict with kernel: Avoid duplicating symbols in the kernel library #34546

To facilitate compiling the external kernel library with different symbol visibility, re-use the same source file list from its dependent internal counterpart.

sedited · 2026-03-14T09:11:35Z

Rebased db85819 -> 1188944 (kernelInternalLib_26 -> kernelInternalLib_27, compare)

Fixed conflict with crypto: Use secure_allocator for AES256_ctx #31774

DrahtBot added the Build system label Oct 20, 2023

DrahtBot added the CI failed label Oct 20, 2023

This was referenced Oct 20, 2023

[WIP] BIP300 (Drivechains) consensus-level logic #28311

Closed

util: Remove DirIsWritable, GetUniquePath #28075

Merged

Get rid of shutdown.cpp/shutdown.h, use SignalInterrupt directly #28051

Merged

sedited force-pushed the kernelInternalLib branch from 3e169c7 to e37fbee Compare October 20, 2023 14:01

sedited force-pushed the kernelInternalLib branch from e37fbee to a5e4960 Compare October 20, 2023 15:00

This was referenced Oct 20, 2023

test: Extend stale_tip_peer_management test #23352

Closed

descriptors: Be able to specify change and receiving in a single descriptor string #22838

Merged

hebasto reviewed Oct 21, 2023

View reviewed changes

src/Makefile.am Outdated Show resolved Hide resolved

sedited force-pushed the kernelInternalLib branch from a5e4960 to d6032c5 Compare October 21, 2023 10:15

sedited force-pushed the kernelInternalLib branch from d6032c5 to f1b1456 Compare October 21, 2023 11:00

sedited force-pushed the kernelInternalLib branch from f1b1456 to 1239bdf Compare October 23, 2023 20:44

DrahtBot mentioned this pull request Oct 23, 2023

type-safe(r) GenTxid constructors #28658

Closed

sedited force-pushed the kernelInternalLib branch from 1239bdf to 04d7a98 Compare October 23, 2023 21:48

DrahtBot mentioned this pull request Nov 18, 2025

policy: allow <minrelay txns in package context if paid for by cpfp #33892

Merged

DrahtBot mentioned this pull request Nov 25, 2025

Replace cluster linearization algorithm with SFL #32545

Merged

This was referenced Dec 4, 2025

Implementation of SwiftSync #34004

Draft

Add util::Expected (std::expected) #34006

Merged

rest: allow reading partial block data from storage #33657

Merged

janb84 reviewed Dec 8, 2025

View reviewed changes

janb84 reviewed Dec 9, 2025

View reviewed changes

sedited mentioned this pull request Dec 10, 2025

doc: clarify libbitcoinkernel usage in libraries design #34042

Draft

This was referenced Dec 11, 2025

crypto: Use secure_allocator for AES256_ctx #31774

Merged

fees: Introduce Mempool Based Fee Estimation to reduce overestimation #34075

Open

janb84 reviewed Dec 16, 2025

View reviewed changes

DrahtBot mentioned this pull request Jan 15, 2026

refactor: [move-only] Merge core_io module, remove from libkernel #34296

Merged

DrahtBot mentioned this pull request Jan 22, 2026

kernel: use structured logging and simplify logging interface #34374

Draft

maflcko mentioned this pull request Feb 2, 2026

kernel: Introduce C header API #30595

Merged

DrahtBot mentioned this pull request Feb 4, 2026

Replace boost signals with minimal compatible implementation #34495

Open

DrahtBot mentioned this pull request Feb 12, 2026

kernel: Avoid duplicating symbols in the kernel library #34546

Merged

This was referenced Feb 22, 2026

node: scale default -dbcache with system RAM #34641

Open

[RFC] BlockMap and CChain Concurrency Improvement #34654

Draft

validation: fetch block inputs on parallel threads #31132

Open

[POC] Full Libevent removal #34411

Draft

Introduce internal kernel library

1188944

To facilitate compiling the external kernel library with different symbol visibility, re-use the same source file list from its dependent internal counterpart.

Conversation

sedited commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DrahtBot commented Oct 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage & Benchmarks

Reviews

Conflicts

Uh oh!

Uh oh!

hebasto commented Oct 21, 2023

Uh oh!

sedited commented Oct 21, 2023

Uh oh!

hebasto commented Oct 21, 2023

Uh oh!

sedited commented Nov 25, 2025

Uh oh!

janb84 left a comment

Choose a reason for hiding this comment

Uh oh!

sedited commented Dec 9, 2025

Uh oh!

janb84 left a comment

Choose a reason for hiding this comment

Uh oh!

maflcko commented Dec 9, 2025

Uh oh!

sedited commented Dec 9, 2025

Uh oh!

yuvicc commented Dec 16, 2025

Uh oh!

janb84 left a comment

Choose a reason for hiding this comment

Uh oh!

sedited commented Jan 13, 2026

Uh oh!

sedited commented Jan 19, 2026

Uh oh!

sedited commented Feb 20, 2026

Uh oh!

sedited commented Mar 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

sedited commented Oct 20, 2023 •

edited

Loading

DrahtBot commented Oct 20, 2023 •

edited

Loading