Skip to content

max_thread_pool_size setting for changing max Threads in Global Thread Pool#11668

Merged
akuzm merged 11 commits intoClickHouse:masterfrom
bharatnc:bnc/config-max-num-threads
Jun 24, 2020
Merged

max_thread_pool_size setting for changing max Threads in Global Thread Pool#11668
akuzm merged 11 commits intoClickHouse:masterfrom
bharatnc:bnc/config-max-num-threads

Conversation

@bharatnc
Copy link
Contributor

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

max_thread_pool_size config for changing the maximum number of Threads in Global Thread Pool.

Detailed description / Documentation draft:

This PR makes the maximum number of threads in the Global Thread Pool configurable. Currently this value defaults to 10000 and is being set inside ThreadPool.cpp.

The setting for this option is max_thread_pool_size and is being added to the config.xml file and is loaded while returning a GlobalThreadPool instance.

Also adds the configuration to relevant tests and docs (en) have also been added.

Addresses the issue: #7099

@blinkov blinkov added doc-alert pr-feature Pull request with new product feature labels Jun 14, 2020
bharatnc added 4 commits June 15, 2020 22:04
This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.
This adds the `max_thread_pool_size` config to
tests/server-test.xml file.
This adds the docs for the new server level setting
`max_thread_pool_size`.
@bharatnc
Copy link
Contributor Author

Can't seem to find the reason why 00092_obfuscator fails consistently with all the tests. Any inputs? Cc: @alexey-milovidov @blinkov

@akuzm akuzm self-assigned this Jun 19, 2020
@tavplubix
Copy link
Member

It's not false-positive, test fails because there is no instance of Poco::Util::Application in clickhouse-obfuscator. Some unit tests probably fail for same reason.

$ clickhouse-obfuscator --structure "URL String, Title String, SearchPhrase String" --input-format TSV --output-format TSV --seed hello < ${CLICKHOUSE_TMP}/data.tsv > ${CLICKHOUSE_TMP}/data1000.tsv
Training models
Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Null pointer: _pInstance in file "../contrib/poco/Util/include/Poco/Util/Application.h", line 457, Stack trace (when copying this message, always include the lines below):

0. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/libcxx/include/exception:129: std::exception::capture() @ 0x15cb1fe8 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
1. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/libcxx/include/exception:109: std::exception::exception() @ 0x15cb1fb5 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
2. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x2190c1b3 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
3. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/poco/Foundation/src/Exception.cpp:128: Poco::LogicException::LogicException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x2190c9ad in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
4. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/poco/Foundation/src/Exception.cpp:130: Poco::NullPointerException::NullPointerException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x2190d10d in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
5. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/poco/Foundation/src/Bugcheck.cpp:42: Poco::Bugcheck::nullPointer(char const*, char const*, int) @ 0x218ff8fd in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
6. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/poco/Util/include/Poco/Util/Application.h:457: Poco::Util::Application::instance() @ 0x15cf1581 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
7. /home/avtokmakov/ch/ClickHouse/build_debug/../src/Common/ThreadPool.cpp:269: GlobalThreadPool::instance() @ 0x15cee03d in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
8. /home/avtokmakov/ch/ClickHouse/build_debug/../src/Common/ThreadPool.h:152: ThreadFromGlobalPool::ThreadFromGlobalPool<DB::ParallelParsingBlockInputStream::ParallelParsingBlockInputStream(DB::ParallelParsingBlockInputStream::Params const&)::'lambda'()>(DB::ParallelParsingBlockInputStream::ParallelParsingBlockInputStream(DB::ParallelParsingBlockInputStream::Params const&)::'lambda'()&&) @ 0x1dd97ca2 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
9. /home/avtokmakov/ch/ClickHouse/build_debug/../src/DataStreams/ParallelParsingBlockInputStream.h:98: DB::ParallelParsingBlockInputStream::ParallelParsingBlockInputStream(DB::ParallelParsingBlockInputStream::Params const&) @ 0x1dd97810 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
10. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/libcxx/include/memory:2214: std::__1::__compressed_pair_elem<DB::ParallelParsingBlockInputStream, 1, false>::__compressed_pair_elem<DB::ParallelParsingBlockInputStream::Params&, 0ul>(std::__1::piecewise_construct_t, std::__1::tuple<DB::ParallelParsingBlockInputStream::Params&>, std::__1::__tuple_indices<0ul>) @ 0x1dd97498 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
11. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/libcxx/include/memory:2299: std::__1::__compressed_pair<std::__1::allocator<DB::ParallelParsingBlockInputStream>, DB::ParallelParsingBlockInputStream>::__compressed_pair<std::__1::allocator<DB::ParallelParsingBlockInputStream>&, DB::ParallelParsingBlockInputStream::Params&>(std::__1::piecewise_construct_t, std::__1::tuple<std::__1::allocator<DB::ParallelParsingBlockInputStream>&>, std::__1::tuple<DB::ParallelParsingBlockInputStream::Params&>) @ 0x1dd97185 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
12. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/libcxx/include/memory:3569: std::__1::__shared_ptr_emplace<DB::ParallelParsingBlockInputStream, std::__1::allocator<DB::ParallelParsingBlockInputStream> >::__shared_ptr_emplace<DB::ParallelParsingBlockInputStream::Params&>(std::__1::allocator<DB::ParallelParsingBlockInputStream>, DB::ParallelParsingBlockInputStream::Params&) @ 0x1dd96e1a in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
13. /home/avtokmakov/ch/ClickHouse/build_debug/../contrib/libcxx/include/memory:4400: std::__1::enable_if<!(is_array<DB::ParallelParsingBlockInputStream>::value), std::__1::shared_ptr<DB::ParallelParsingBlockInputStream> >::type std::__1::make_shared<DB::ParallelParsingBlockInputStream, DB::ParallelParsingBlockInputStream::Params&>(DB::ParallelParsingBlockInputStream::Params&) @ 0x1dd948f0 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
14. /home/avtokmakov/ch/ClickHouse/build_debug/../src/Formats/FormatFactory.cpp:190: DB::FormatFactory::getInput(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ReadBuffer&, DB::Block const&, DB::Context const&, unsigned long, std::__1::function<void ()>) const @ 0x1dd916bb in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
15. /home/avtokmakov/ch/ClickHouse/build_debug/../src/Interpreters/Context.cpp:1903: DB::Context::getInputFormat(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::ReadBuffer&, DB::Block const&, unsigned long) const @ 0x1cbfa175 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
16. /home/avtokmakov/ch/ClickHouse/build_debug/../programs/obfuscator/Obfuscator.cpp:1107: mainEntryClickHouseObfuscator(int, char**) @ 0x15ecbb4d in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
17. /home/avtokmakov/ch/ClickHouse/build_debug/../programs/main.cpp:324: main @ 0x15c9d385 in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
18. /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344: __libc_start_main @ 0x21b97 in /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.27.so
19. _start @ 0x15c9d02a in /home/avtokmakov/ch/ClickHouse/build_debug/programs/clickhouse
 (version 20.5.1.1)

@bharatnc
Copy link
Contributor Author

Thank you @akuzm for your help with fixing the failing test.

@akuzm
Copy link
Contributor

akuzm commented Jun 24, 2020

Thank you @akuzm for your help with fixing the failing test.

You're welcome.

'Yandex third-party checks' are broken for a different reason, will merge now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature Pull request with new product feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants