fuzz: Add fuzzing harness for LoadMempool(...) and DumpMempool(...) by practicalswift · Pull Request #19259 · bitcoin/bitcoin

practicalswift · 2020-06-12T14:35:30Z

Add fuzzing harness for LoadMempool(...) and DumpMempool(...).

See doc/fuzzing.md for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the Bitcoin Core fuzzing corpus repo.

Happy fuzzing :)

DrahtBot · 2020-06-13T02:51:17Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

tests: Add fuzzing harness for versionbits #21380 (versionbits: Refactor and add fuzzing harness by ajtowns)
Speedy trial support for versionbits #21377 (Speedy trial support for versionbits by ajtowns)
Move GetDataDir to ArgsManager #21244 (Move GetDataDir to ArgsManager by kiminuo)
fuzz: Add tx_pool fuzz target #21142 (fuzz: Add tx_pool fuzz target by MarcoFalke)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Crypt-iQ · 2020-08-18T13:48:16Z

I know this is sort of in-progress as there are outstanding UBSan warnings, but this won't compile on my macOS v10.15.4, clang 10.0.1. Needs a rebase I think.

Making all in src
  CXX      test/fuzz/addition_overflow-addition_overflow.o
In file included from test/fuzz/addition_overflow.cpp:7:
./test/fuzz/util.h:347:13: error: no matching function for call to 'AdditionOverflow'
        if (AdditionOverflow((uint64_t)fuzzed_file->m_offset, random_bytes.size())) {
            ^~~~~~~~~~~~~~~~
./test/fuzz/util.h:201:16: note: candidate template ignored: deduced conflicting types for parameter 'T' ('unsigned long long' vs. 'unsigned long')
NODISCARD bool AdditionOverflow(const T i, const T j) noexcept
               ^
./test/fuzz/util.h:359:13: error: no matching function for call to 'AdditionOverflow'
        if (AdditionOverflow(fuzzed_file->m_offset, n)) {
            ^~~~~~~~~~~~~~~~
./test/fuzz/util.h:201:16: note: candidate template ignored: deduced conflicting types for parameter 'T' ('long long' vs. 'long')
NODISCARD bool AdditionOverflow(const T i, const T j) noexcept
               ^
2 errors generated.
make[2]: *** [test/fuzz/addition_overflow-addition_overflow.o] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

practicalswift · 2020-08-18T18:58:32Z

@Crypt-iQ Oh, thanks for letting me know. Now rebased and hopefully no compilation errors? :)

Crypt-iQ · 2020-08-19T02:34:50Z

@practicalswift now builds on my macOS :)

practicalswift · 2021-03-01T11:44:33Z

Rebase number 12 completed :)

Now using the recently introduced MakeNoLogFileContext which wasn't around when this fuzz testing PR was submitted nine months ago.

At nine months PR pregnancy: is this fuzz testing PR baby getting ready to meet the world (master)? :)

jonatack

Light code review ACK d9b0d15607167cbfef52bcc73b964f1201b71796 and ran the fuzzer

$ FUZZ=validation_load_mempool src/test/fuzz/fuzz 
INFO: Seed: 2149651356
INFO: Loaded 1 modules   (643553 inline 8-bit counters): 643553 [0x5603d1a48d48, 0x5603d1ae5f29), 
INFO: Loaded 1 PC tables (643553 PCs): 643553 [0x5603d1ae5f30,0x5603d24b7d40), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED cov: 2470 ft: 2469 corp: 1/1b exec/s: 0 rss: 194Mb
#5	NEW    cov: 2470 ft: 2476 corp: 2/5b lim: 4 exec/s: 0 rss: 194Mb L: 4/4 MS: 3 CopyPart-CopyPart-CrossOver-
#8	NEW    cov: 2470 ft: 2483 corp: 3/7b lim: 4 exec/s: 0 rss: 195Mb L: 2/4 MS: 3 ChangeByte-ChangeBit-CopyPart-
.../...
#3035373	REDUCE cov: 12335 ft: 46498 corp: 1039/729Kb lim: 4096 exec/s: 1489 rss: 518Mb L: 1711/4096 MS: 2 ChangeByte-EraseBytes-
#3036089	REDUCE cov: 12335 ft: 46498 corp: 1039/729Kb lim: 4096 exec/s: 1489 rss: 518Mb L: 132/4096 MS: 1 EraseBytes-
#3036186	REDUCE cov: 12335 ft: 46498 corp: 1039/729Kb lim: 4096 exec/s: 1489 rss: 518Mb L: 291/4096 MS: 2 ChangeByte-EraseBytes-

I'm not sure if there is a recent regression (unrelated to this patch) in our fuzzing utils or in a recent apt update on Debian testing or if I'm just doing something wrong, but like with the I2P fuzzing PR, I'm seeing OOM with qa-assets.

fuzz output

$ FUZZ=validation_load_mempool src/test/fuzz/fuzz ../qa-assets/fuzz_seed_corpus
INFO: Seed: 158284216
INFO: Loaded 1 modules   (643553 inline 8-bit counters): 643553 [0x5590b92c0d48, 0x5590b935df29), 
INFO: Loaded 1 PC tables (643553 PCs): 643553 [0x5590b935df30,0x5590b9d2fd40), 
INFO:   237105 files found in ../qa-assets/fuzz_seed_corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
INFO: seed corpus: files: 237105 min: 1b max: 3986616b total: 4147898382b rss: 380Mb
#4096	pulse  cov: 2471 ft: 2489 corp: 5/15b exec/s: 2048 rss: 497Mb
#8192	pulse  cov: 2513 ft: 2578 corp: 14/83b exec/s: 2048 rss: 589Mb
#16384	pulse  cov: 2712 ft: 3173 corp: 23/175b exec/s: 2730 rss: 655Mb
#32768	pulse  cov: 2715 ft: 3207 corp: 32/307b exec/s: 2978 rss: 681Mb
#65536	pulse  cov: 4935 ft: 6993 corp: 42/695b exec/s: 3120 rss: 681Mb
#131072	pulse  cov: 5699 ft: 12269 corp: 53/1944b exec/s: 3196 rss: 681Mb
==13249== ERROR: libFuzzer: out-of-memory (used: 2160Mb; limit: 2048Mb)
   To change the out-of-memory limit use -rss_limit_mb=<N>

Live Heap Allocations: 93237649 bytes in 253456 chunks; quarantined: 249855423 bytes in 15591 chunks; 1526469 other chunks; total chunks: 1795516; showing top 95% (at most 8 unique contexts)
22440640 byte(s) (24%) in 237105 allocation(s)
    #0 0x5590b517660d in malloc (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2d5760d)
    #1 0x5590b5088ef7 in operator new(unsigned long) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c69ef7)
    #2 0x5590b50a0bbc in fuzzer::ReadCorpora(std::Fuzzer::vector<std::Fuzzer::basic_string<char, std::Fuzzer::char_traits<char>, std::Fuzzer::allocator<char> >, fuzzer::fuzzer_allocator<std::Fuzzer::basic_string<char, std::Fuzzer::char_traits<char>, std::Fuzzer::allocator<char> > > > const&, std::Fuzzer::vector<std::Fuzzer::basic_string<char, std::Fuzzer::char_traits<char>, std::Fuzzer::allocator<char> >, fuzzer::fuzzer_allocator<std::Fuzzer::basic_string<char, std::Fuzzer::char_traits<char>, std::Fuzzer::allocator<char> > > > const&) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c81bbc)
    #3 0x5590b50a0782 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c81782)
    #4 0x5590b50c9c22 in main (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2caac22)
    #5 0x7ff6cc2bbd09 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26d09)

21499328 byte(s) (23%) in 12 allocation(s)
    #0 0x5590b517660d in malloc (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2d5760d)
    #1 0x5590b5088ef7 in operator new(unsigned long) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c69ef7)
    #2 0x5590b50c9c22 in main (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2caac22)
    #3 0x7ff6cc2bbd09 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26d09)

16777216 byte(s) (17%) in 1 allocation(s)
    #0 0x5590b51a5d4d in operator new(unsigned long) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2d86d4d)
    #1 0x5590b52a03b5 in __gnu_cxx::new_allocator<uint256>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/ext/new_allocator.h:115:27
    #2 0x5590b52a03b5 in std::allocator_traits<std::allocator<uint256> >::allocate(std::allocator<uint256>&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/alloc_traits.h:460:20
    #3 0x5590b52a03b5 in std::_Vector_base<uint256, std::allocator<uint256> >::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/stl_vector.h:346:20
    #4 0x5590b5bb7d1a in std::vector<uint256, std::allocator<uint256> >::_M_default_append(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/vector.tcc:635:34
    #5 0x5590b5bb7872 in std::vector<uint256, std::allocator<uint256> >::resize(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/stl_vector.h:940:4
    #6 0x5590b5bb70c3 in CuckooCache::cache<uint256, SignatureCacheHasher>::setup(unsigned int) /home/jon/projects/bitcoin/bitcoin/src/./cuckoocache.h:344:15
    #7 0x5590b5ddce16 in CuckooCache::cache<uint256, SignatureCacheHasher>::setup_bytes(unsigned long) /home/jon/projects/bitcoin/bitcoin/src/./cuckoocache.h:368:16
    #8 0x5590b5ddce16 in InitScriptExecutionCache() /home/jon/projects/bitcoin/bitcoin/src/validation.cpp:1468:44
    #9 0x5590b64f7a5f in BasicTestingSetup::BasicTestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/test/util/setup_common.cpp:110:5
    #10 0x5590b64f9599 in ChainTestingSetup::ChainTestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/test/util/setup_common.cpp:130:7
    #11 0x5590b64fbd3f in TestingSetup::TestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/test/util/setup_common.cpp:169:7
    #12 0x5590b52e7bf7 in std::_MakeUniq<TestingSetup const>::__single_object std::make_unique<TestingSetup const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/unique_ptr.h:962:34
    #13 0x5590b52dde12 in std::unique_ptr<TestingSetup const, std::default_delete<TestingSetup const> > MakeNoLogFileContext<TestingSetup const>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/./test/util/setup_common.h:170:12
    #14 0x5590b5665353 in initialize_validation_load_mempool() /home/jon/projects/bitcoin/bitcoin/src/test/fuzz/validation_load_mempool.cpp:28:39
    #15 0x5590b51ac91c in void std::__invoke_impl<void, void (*&)()>(std::__invoke_other, void (*&)()) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:60:14
    #16 0x5590b51ac91c in std::enable_if<is_invocable_r_v<void, void (*&)()>, void>::type std::__invoke_r<void, void (*&)()>(void (*&)()) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:110:2
    #17 0x5590b51ac91c in std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/std_function.h:291:9
    #18 0x5590b579287c in std::function<void ()>::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/std_function.h:622:14
    #19 0x5590b6b2b312 in initialize() /home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz.cpp:44:5
    #20 0x5590b6b2c84d in LLVMFuzzerInitialize /home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz.cpp:70:5
    #21 0x5590b509e9bc in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c7f9bc)
    #22 0x5590b50c9c22 in main (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2caac22)
    #23 0x7ff6cc2bbd09 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26d09)

16777216 byte(s) (17%) in 1 allocation(s)
    #0 0x5590b51a5d4d in operator new(unsigned long) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2d86d4d)
    #1 0x5590b52a03b5 in __gnu_cxx::new_allocator<uint256>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/ext/new_allocator.h:115:27
    #2 0x5590b52a03b5 in std::allocator_traits<std::allocator<uint256> >::allocate(std::allocator<uint256>&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/alloc_traits.h:460:20
    #3 0x5590b52a03b5 in std::_Vector_base<uint256, std::allocator<uint256> >::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/stl_vector.h:346:20
    #4 0x5590b5bb7d1a in std::vector<uint256, std::allocator<uint256> >::_M_default_append(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/vector.tcc:635:34
    #5 0x5590b5bb7872 in std::vector<uint256, std::allocator<uint256> >::resize(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/stl_vector.h:940:4
    #6 0x5590b5bb70c3 in CuckooCache::cache<uint256, SignatureCacheHasher>::setup(unsigned int) /home/jon/projects/bitcoin/bitcoin/src/./cuckoocache.h:344:15
    #7 0x5590b5bb4862 in CuckooCache::cache<uint256, SignatureCacheHasher>::setup_bytes(unsigned long) /home/jon/projects/bitcoin/bitcoin/src/./cuckoocache.h:368:16
    #8 0x5590b5bb4862 in (anonymous namespace)::CSignatureCache::setup_bytes(unsigned long) /home/jon/projects/bitcoin/bitcoin/src/script/sigcache.cpp:80:25
    #9 0x5590b5bb4862 in InitSignatureCache() /home/jon/projects/bitcoin/bitcoin/src/script/sigcache.cpp:100:36
    #10 0x5590b64f7a4f in BasicTestingSetup::BasicTestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/test/util/setup_common.cpp:109:5
    #11 0x5590b64f9599 in ChainTestingSetup::ChainTestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/test/util/setup_common.cpp:130:7
    #12 0x5590b64fbd3f in TestingSetup::TestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/test/util/setup_common.cpp:169:7
    #13 0x5590b52e7bf7 in std::_MakeUniq<TestingSetup const>::__single_object std::make_unique<TestingSetup const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/unique_ptr.h:962:34
    #14 0x5590b52dde12 in std::unique_ptr<TestingSetup const, std::default_delete<TestingSetup const> > MakeNoLogFileContext<TestingSetup const>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/./test/util/setup_common.h:170:12
    #15 0x5590b5665353 in initialize_validation_load_mempool() /home/jon/projects/bitcoin/bitcoin/src/test/fuzz/validation_load_mempool.cpp:28:39
    #16 0x5590b51ac91c in void std::__invoke_impl<void, void (*&)()>(std::__invoke_other, void (*&)()) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:60:14
    #17 0x5590b51ac91c in std::enable_if<is_invocable_r_v<void, void (*&)()>, void>::type std::__invoke_r<void, void (*&)()>(void (*&)()) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:110:2
    #18 0x5590b51ac91c in std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/std_function.h:291:9
    #19 0x5590b579287c in std::function<void ()>::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/std_function.h:622:14
    #20 0x5590b6b2b312 in initialize() /home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz.cpp:44:5
    #21 0x5590b6b2c84d in LLVMFuzzerInitialize /home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz.cpp:70:5
    #22 0x5590b509e9bc in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c7f9bc)
    #23 0x5590b50c9c22 in main (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2caac22)
    #24 0x7ff6cc2bbd09 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26d09)

8388608 byte(s) (8%) in 1 allocation(s)
    #0 0x5590b517660d in malloc (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2d5760d)
    #1 0x5590b5088ef7 in operator new(unsigned long) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c69ef7)
    #2 0x5590b50ac917 in fuzzer::GetSizedFilesFromDir(std::Fuzzer::basic_string<char, std::Fuzzer::char_traits<char>, std::Fuzzer::allocator<char> > const&, std::Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >*) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c8d917)
    #3 0x5590b50a0bbc in fuzzer::ReadCorpora(std::Fuzzer::vector<std::Fuzzer::basic_string<char, std::Fuzzer::char_traits<char>, std::Fuzzer::allocator<char> >, fuzzer::fuzzer_allocator<std::Fuzzer::basic_string<char, std::Fuzzer::char_traits<char>, std::Fuzzer::allocator<char> > > > const&, std::Fuzzer::vector<std::Fuzzer::basic_string<char, std::Fuzzer::char_traits<char>, std::Fuzzer::allocator<char> >, fuzzer::fuzzer_allocator<std::Fuzzer::basic_string<char, std::Fuzzer::char_traits<char>, std::Fuzzer::allocator<char> > > > const&) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c81bbc)
    #4 0x5590b50a0782 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c81782)
    #5 0x5590b50c9c22 in main (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2caac22)
    #6 0x7ff6cc2bbd09 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26d09)

1294000 byte(s) (1%) in 1 allocation(s)
    #0 0x5590b51a5d4d in operator new(unsigned long) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2d86d4d)
    #1 0x5590b5376cf5 in __gnu_cxx::new_allocator<unsigned long>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/ext/new_allocator.h:115:27
    #2 0x5590b5376cf5 in std::allocator_traits<std::allocator<unsigned long> >::allocate(std::allocator<unsigned long>&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/alloc_traits.h:460:20
    #3 0x5590b5376cf5 in std::_Vector_base<unsigned long, std::allocator<unsigned long> >::_M_allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/stl_vector.h:346:20
    #4 0x5590b60f606b in std::vector<unsigned long, std::allocator<unsigned long> >::_M_default_append(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/vector.tcc:635:34
    #5 0x5590b60f3042 in std::vector<unsigned long, std::allocator<unsigned long> >::resize(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/stl_vector.h:940:4
    #6 0x5590b60f2c2b in CRollingBloomFilter::CRollingBloomFilter(unsigned int, double) /home/jon/projects/bitcoin/bitcoin/src/bloom.cpp:196:10
    #7 0x5590b66056cb in (anonymous namespace)::PeerManagerImpl::PeerManagerImpl(CChainParams const&, CConnman&, BanMan*, CScheduler&, ChainstateManager&, CTxMemPool&, bool) /home/jon/projects/bitcoin/bitcoin/src/net_processing.cpp:1229:29
    #8 0x5590b66056cb in std::_MakeUniq<(anonymous namespace)::PeerManagerImpl>::__single_object std::make_unique<(anonymous namespace)::PeerManagerImpl, CChainParams const&, CConnman&, BanMan*&, CScheduler&, ChainstateManager&, CTxMemPool&, bool&>(CChainParams const&, CConnman&, BanMan*&, CScheduler&, ChainstateManager&, CTxMemPool&, bool&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/unique_ptr.h:962:34
    #9 0x5590b660412f in PeerManager::make(CChainParams const&, CConnman&, BanMan*, CScheduler&, ChainstateManager&, CTxMemPool&, bool) /home/jon/projects/bitcoin/bitcoin/src/net_processing.cpp:1213:12
    #10 0x5590b64fc949 in TestingSetup::TestingSetup(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/test/util/setup_common.cpp:193:22
    #11 0x5590b52e7bf7 in std::_MakeUniq<TestingSetup const>::__single_object std::make_unique<TestingSetup const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/unique_ptr.h:962:34
    #12 0x5590b52dde12 in std::unique_ptr<TestingSetup const, std::default_delete<TestingSetup const> > MakeNoLogFileContext<TestingSetup const>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<char const*, std::allocator<char const*> > const&) /home/jon/projects/bitcoin/bitcoin/src/./test/util/setup_common.h:170:12
    #13 0x5590b5665353 in initialize_validation_load_mempool() /home/jon/projects/bitcoin/bitcoin/src/test/fuzz/validation_load_mempool.cpp:28:39
    #14 0x5590b51ac91c in void std::__invoke_impl<void, void (*&)()>(std::__invoke_other, void (*&)()) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:60:14
    #15 0x5590b51ac91c in std::enable_if<is_invocable_r_v<void, void (*&)()>, void>::type std::__invoke_r<void, void (*&)()>(void (*&)()) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/invoke.h:110:2
    #16 0x5590b51ac91c in std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/std_function.h:291:9
    #17 0x5590b579287c in std::function<void ()>::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/std_function.h:622:14
    #18 0x5590b6b2b312 in initialize() /home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz.cpp:44:5
    #19 0x5590b6b2c84d in LLVMFuzzerInitialize /home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz.cpp:70:5
    #20 0x5590b509e9bc in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c7f9bc)
    #21 0x5590b50c9c22 in main (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2caac22)
    #22 0x7ff6cc2bbd09 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26d09)

1049024 byte(s) (1%) in 2 allocation(s)
    #0 0x5590b517660d in malloc (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2d5760d)
    #1 0x5590b6d71072 in checked_malloc /home/jon/projects/bitcoin/bitcoin/src/secp256k1/./src/util.h:92:17
    #2 0x5590b6d71072 in secp256k1_context_create /home/jon/projects/bitcoin/bitcoin/src/secp256k1/src/secp256k1.c:153:50

1048576 byte(s) (1%) in 1 allocation(s)
    #0 0x5590b517660d in malloc (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2d5760d)
    #1 0x5590b5088ef7 in operator new(unsigned long) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c69ef7)
    #2 0x5590b50b2669 in fuzzer::Fuzzer::Loop(std::Fuzzer::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c93669)
    #3 0x5590b50a07d8 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2c817d8)
    #4 0x5590b50c9c22 in main (/home/jon/projects/bitcoin/bitcoin/src/test/fuzz/fuzz+0x2caac22)
    #5 0x7ff6cc2bbd09 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x26d09)

MS: 0 ; base unit: 0000000000000000000000000000000000000000


artifact_prefix='./'; Test unit written to ./oom-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64: 
SUMMARY: libFuzzer: out-of-memory

src/validation.cpp

src/test/fuzz/util.h

…_END).

…/fuzz/mock

practicalswift · 2021-03-11T22:46:11Z

@jonatack

Thanks a lot for reviewing!

All feedback addressed (+ added using FopenFn for readability): this PR should hopefully be ready for final review :)

Crypt-iQ · 2021-03-12T13:23:22Z

@jonatack libfuzzer is in-process fuzzing with an in-memory corpus and I have noticed on my Arch box, that more inputs and larger inputs will make rss increase significantly. In the crash log, I notice that the max length for inputs is 1MB. Just a thought.

jonatack · 2021-03-12T13:35:12Z

@Crypt-iQ yes, for me this is a new issue (after a couple years of testing fuzz patches on this project) and only with qa-assets.

jonatack · 2021-03-12T15:06:03Z

re-ACK f00061202867dd965fe8ca51253d1600776f8ab0 per git range-diff e0bc27a d9b0d15 f000612, looked over the changes again, fuzz build + ran fuzzer, debug non-fuzz build + ran/halted bitcoind a couple times on mainnet/signet, loading/dumping mempool seemed nominal

Thanks for adding the FopenFn type alias. It's a nice improvement.

maflcko · 2021-03-12T15:28:08Z

src/test/fuzz/validation_load_mempool.cpp

Could avoid this namespace by making fuzzed_fopen a lambda with smallest scope possible?

Good point! Now addressed.

maflcko · 2021-03-12T15:32:09Z

src/test/fuzz/validation_load_mempool.cpp

would be nice to not introduce a new global

Good point! Now addressed.

Not addressed?

nvm. @dongcarl will fix this 😬

I thought you meant the introduction of fuzzed_fopen? What did you mean? :)

Oh, the ::ChainstateActive()

jonatack · 2021-03-15T17:22:09Z

Tested re-ACK 68afd3e

Nice improvement.

git diff f000612 68afd3e

diff --git a/src/test/fuzz/validation_load_mempool.cpp b/src/test/fuzz/validation_load_mempool.cpp
index 97e705ef30..e1a21b6c53 100644
--- a/src/test/fuzz/validation_load_mempool.cpp
+++ b/src/test/fuzz/validation_load_mempool.cpp
@@ -14,15 +14,6 @@
 #include <cstdint>
 #include <vector>
 
-namespace {
-FuzzedFileProvider* fuzzed_file_provider_ptr = nullptr;
-
-FILE* fuzzed_fopen(const fs::path&, const char*)
-{
-    return fuzzed_file_provider_ptr->open();
-}
-} // namespace
-
 void initialize_validation_load_mempool()
 {
     static const auto testing_setup = MakeNoLogFileContext<const TestingSetup>();
@@ -33,11 +24,11 @@ FUZZ_TARGET_INIT(validation_load_mempool, initialize_validation_load_mempool)
     FuzzedDataProvider fuzzed_data_provider{buffer.data(), buffer.size()};
     SetMockTime(ConsumeTime(fuzzed_data_provider));
     FuzzedFileProvider fuzzed_file_provider = ConsumeFile(fuzzed_data_provider);
-    fuzzed_file_provider_ptr = &fuzzed_file_provider;
 
     CTxMemPool pool{};
+    auto fuzzed_fopen = [&](const fs::path&, const char*) {
+        return fuzzed_file_provider.open();
+    };
     (void)LoadMempool(pool, ::ChainstateActive(), fuzzed_fopen);
     (void)DumpMempool(pool, fuzzed_fopen, true);
-
-    fuzzed_file_provider_ptr = nullptr;
 }

…nd DumpMempool(...) 68afd3e tests: Add fuzzing harness for LoadMempool(...) and DumpMempool(...) (practicalswift) 91af6b9 validation: Make DumpMempool(...) and LoadMempool(...) easier to test/fuzz/mock (practicalswift) af322c7 tests: Set errno in FuzzedFileProvider. Implement seek(..., ..., SEEK_END). (practicalswift) Pull request description: Add fuzzing harness for `LoadMempool(...)` and `DumpMempool(...)`. See [`doc/fuzzing.md`](https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md) for information on how to fuzz Bitcoin Core. Don't forget to contribute any coverage increasing inputs you find to the [Bitcoin Core fuzzing corpus repo](https://github.com/bitcoin-core/qa-assets). Happy fuzzing :) ACKs for top commit: jonatack: Tested re-ACK 68afd3e Tree-SHA512: 4b5fcaa87e6eb478611d3b68eb6859645a5e121e7e3b056ad2815699dace0a6123706ff542def371b47f4ab3ce2b8a29782026d84fb505827121e9b4cc7dac31

DrahtBot added Build system Tests Validation labels Jun 12, 2020

practicalswift force-pushed the fuzzers-mempool-io branch from 105277a to 197ba1b Compare June 12, 2020 14:47

maflcko removed Build system Validation labels Jun 12, 2020

This was referenced Jun 16, 2020

Three UBSan warnings when loading corrupt mempool.dat files #19278

Closed

Add seeds bitcoin-core/qa-assets#17

Merged

This was referenced Jun 25, 2020

Loadmempool fuzz corpus bitcoin-core/qa-assets#18

Closed

Fix UBSan warnings triggered when loading corrupt mempool.dat files #19381

Closed

DrahtBot mentioned this pull request Jun 30, 2020

net: Make DNS lookup mockable, add fuzzing harness #19415

Merged

DrahtBot added the Needs rebase label Jul 11, 2020

practicalswift force-pushed the fuzzers-mempool-io branch from 197ba1b to 80a379a Compare July 11, 2020 18:23

DrahtBot removed the Needs rebase label Jul 11, 2020

DrahtBot mentioned this pull request Jul 11, 2020

Refactor mempool.dat to be extensible, and store missing info #19488

Closed

DrahtBot added the Needs rebase label Jul 18, 2020

practicalswift force-pushed the fuzzers-mempool-io branch 2 times, most recently from 6384bf7 to e07deaf Compare July 18, 2020 11:27

DrahtBot removed the Needs rebase label Jul 18, 2020

practicalswift force-pushed the fuzzers-mempool-io branch from e07deaf to c0c5177 Compare August 18, 2020 18:57

maflcko mentioned this pull request Sep 6, 2020

Remove mempool global #19556

Merged

practicalswift force-pushed the fuzzers-mempool-io branch from a11098c to c309452 Compare February 11, 2021 08:42

DrahtBot added the Needs rebase label Feb 20, 2021

practicalswift force-pushed the fuzzers-mempool-io branch 2 times, most recently from 4af0132 to d99932d Compare February 22, 2021 16:37

DrahtBot removed the Needs rebase label Feb 22, 2021

DrahtBot mentioned this pull request Feb 27, 2021

Move GetDataDir to ArgsManager #21244

Merged

practicalswift force-pushed the fuzzers-mempool-io branch from d99932d to 9589abb Compare March 1, 2021 11:36

practicalswift force-pushed the fuzzers-mempool-io branch from 9589abb to d9b0d15 Compare March 3, 2021 15:03

jonatack reviewed Mar 11, 2021

View reviewed changes

src/validation.cpp Outdated Show resolved Hide resolved

src/test/fuzz/util.h Outdated Show resolved Hide resolved

practicalswift added 2 commits March 11, 2021 22:34

tests: Set errno in FuzzedFileProvider. Implement seek(..., ..., SEEK…

af322c7

…_END).

validation: Make DumpMempool(...) and LoadMempool(...) easier to test…

91af6b9

…/fuzz/mock

practicalswift force-pushed the fuzzers-mempool-io branch from d9b0d15 to f000612 Compare March 11, 2021 22:41

maflcko reviewed Mar 12, 2021

View reviewed changes

tests: Add fuzzing harness for LoadMempool(...) and DumpMempool(...)

68afd3e

practicalswift force-pushed the fuzzers-mempool-io branch from f000612 to 68afd3e Compare March 15, 2021 15:49

maflcko merged commit 67ec26c into bitcoin:master Mar 15, 2021

practicalswift deleted the fuzzers-mempool-io branch April 10, 2021 19:44

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

Conversation

practicalswift commented Jun 12, 2020

Uh oh!

DrahtBot commented Jun 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Conflicts

Uh oh!

Crypt-iQ commented Aug 18, 2020

Uh oh!

practicalswift commented Aug 18, 2020

Uh oh!

Crypt-iQ commented Aug 19, 2020

Uh oh!

practicalswift commented Mar 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonatack left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

practicalswift commented Mar 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Crypt-iQ commented Mar 12, 2021

Uh oh!

jonatack commented Mar 12, 2021

Uh oh!

jonatack commented Mar 12, 2021

Uh oh!

maflcko Mar 12, 2021

Choose a reason for hiding this comment

Uh oh!

practicalswift Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

maflcko Mar 12, 2021

Choose a reason for hiding this comment

Uh oh!

practicalswift Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

maflcko Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

practicalswift Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

maflcko Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

jonatack commented Mar 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

DrahtBot commented Jun 13, 2020 •

edited

Loading

practicalswift commented Mar 1, 2021 •

edited

Loading

practicalswift commented Mar 11, 2021 •

edited

Loading