Fixes: ClickHouse#11163

…cess rights. (#11963)

* Simple github hook

* Add concurrent benchmark to performance test

After the main test, run queries from `website.xml` in parallel using
`clickhouse-benchmark`. This can be useful to test the effects of
concurrency on performance. Comparison test can miss some effects
because it always runs queries sequentially, and many of them are even
single-threaded.

* typo

* fixup

* add flag to continue on errors

* fixup

* fixup

* Compilable setColumns, setConstraints, setIndices

* Working setColumns, setConstraints, setIndices

* Move set*Key methods to StorageInMemoryMetadata

* Working set*Keys methods

* TTL methods in StorageInMemoryMetadata

* All set methods in metadata

* Metadata in read and write methods of IStorage

* make max global thread pool setting configurable

This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.

* add max_thread_pool_size setting to tests

This adds the `max_thread_pool_size` config to
tests/server-test.xml file.

* docs for max_thread_pool_size

This adds the docs for the new server level setting
`max_thread_pool_size`.

* remove extra vertical space

* fixup

* Copy some methods to metdata

* Get column dependencies in StorageInMemoryMetadata

* Move getSampleBlockNonMaterialized to StorageInMemoryMetadata

* getSampleBlockWithVirtuals in StorageInMemoryMetadata

* Compileable getSampleBlockWithColumns in StorageInMemoryMetadata

* Seems to be working getSampleBlockWithColumns in StorageInMemoryMetadata

* Buildable getSampleBlock in StorageInMemoryMetadata

* getConstraints() in StorageInMemoryMetadata (suspicious commit, but pretend to work)

* Fix race condition

* Secondary indices in StorageInMemoryMetadata

* Partition key in StorageInMemoryMetadata

* Compilable sorting key in metadata

* Fix race condition

* Sampling key in StorageInMemoryMetadata

* Primary key in storage metadata

* Missed change for primary key

* TTL in storage in memory metadata

* Settings changes in StorageInMemoryMetadata

* Select query in metadata

* Check methods in metadata

* Better getVirtuals method

* getColumns in StorageInMemoryMetadta (only compilable)

* Fix segmentation fault

* Fix build

* Fix crash

* Fix storage merge

* Truncate with metadata

* Fix SystemTables

* Really atomic metadata

* Fix style

* Fix storage buffer metadata

* Better locks in Storages

* Less locks

* Fix style

* Remove unused method

* Remove unused method better

* Remove int contention

* Disable optimize_skip_unused_shards if sharding_key has non-deterministic func

Example of such functions is rand()

And this patch disables only optimize_skip_unused_shards, i.e. INSERT
code path does not changed, so it will work as before.

* optimize_skip_unused_shards=2 will disable it for nested distributed queries

P.S. Looks like settings can be converted between SettingUInt64 and
SettingBool without breaking binary protocol.

FWIW maybe it is a good idea to change the semantics of the settings as
follow (but I guess that changing semantic is not a good idea, better to
add new settings and deprecate old ones):
- optimize_skip_unused_shards -- accept nesting level on which the
  optimization will work
- force_skip_optimize_shards_nesting -- accept nesting level on which
  the optimization will work

* Add logging of adjusting conditional settings for distributed queries

* Add missing DROP TABLE in 01319_mv_constants_bug

* Improve 01319_optimize_skip_unused_shards_no_nested

Before there is no check that optimize_skip_unused_shards was working
for the first level, use cluster with unavalable shard to guarantee
this.

* Fix locks

* Add missed check

* Move partition key initialization into registerMergeTree

* Remove unused constructor and rename method

* Better naming

* Better logging in storages

* Remove unused variable

* report the number of errors

* fixup

* fixup

* Send logs on fatal errors (development)

* Set send_logs_level to fatal by default

* Update tests

* Turn off traps

* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* Fix memory accounting via HTTP interface

function perf_test()
{
    time yes '127.1:8123/?wait_end_of_query=1' | head -n10000 | xargs -P10000 curl -s -d 'select 1' | grep -x -c 1
}
function server()
{
    local limit=$1 && shift
    clickhouse-server "$@" -- --max_server_memory_usage=$limit
}

perf_test:

- before this patch with 1G  limit: succeed only ~800  queries
- after  this patch with 1G  limit: succeed      ~8000 queries

- before this patch with 10G limit: succeed only ~3000  queries
- after  this patch with 10G limit: succeed      ~10000 queries

Fixes: #11153

* Add a test for memory tracking via HTTP interface

* Update executeQuery.cpp

* Make LIVE VIEW test less wrong (it was depending on the timing of progress messages)

* Do not run 01238_http_memory_tracking under sanitizers

* Tune 01238_http_memory_tracking to make it more optimal

w/o the patch the test fails with, since it succeed only 512 queries.

* Merge with master

* Ad tests with long select and simulatenous alter

* Better comments

* Fix 01213_alter_rename_column_zookeeper

* report

* Fix bad test for LIVE VIEW

* Disable MySQL tests under tsan

* Better 00980_alter_settings test

* Support sumWithOverflow as SimpleAggregateFunction

* Clang-tidy fixes

* Fix header for nullable prewhere column.

* Update tests.

* Fix 01091_num_threads flaps (by adding unique identifier to the queries)

Actually there were two issues:
- missing \n, so it picked the wrong query anway
- no unique identifier, hence it may works incorrectly (and actually,
  event clickhouse-test script executes 'SELECT 1' query)

* Initialize GlobalThreadPool explicitly

* Do not render progress too much

* Stop tests on fatal errors

* jemalloc: fix HAVE_CPU_SPINWAIT for linux aarch64

Fixes: #11163

* Style

* constexpr intExp10

* Fix bugs in DateTime64 parsing

* Added a test

* Fix wrong code

* Fix strange code

* Update tests

* jemalloc: jemalloc_preamble.h is common

* jemalloc: fix JEMALLOC_NOTHROW/JEMALLOC_CXX_THROW for freebsd/darwin

* jemalloc: add freebsd pregenerated header

* jemalloc: add darwin pregenerated header

* jemalloc: enable by default for freebsd/darwin

* jemalloc: drop superfluous include directory

* jemalloc: explicitly disable pthread_setname_np on freebsd (requires 12.1)

* Add drop replica alter support

* fix merge state

Fix style && build

* Add tests and docs

Fix integration-tests

* only drop inactive replica

update doc

Increase timeout to release the zookeeper Ephemeral nodes

Fix code comment

use PartitionManager

make integrations test passed

* add SYSTEM DROP REPLICA

* fix: can not drop local table and replace ALTER with SYSTEM

fix test_drop_replica

fix drop replica '/path/to/zk/' ending in '/' and update doc

* add drop relica database and the whole replica

fix removeReplicaByZKPath

fix bug: add zkpath empty judge

fix: rewrite code

delete useless code.

fix:ast fromat

fix bug

add test_drop_replica

add drop_replica doc

add drop databse checkAccess

refactor dropReplica

update tests

add static method StorageReplicatedMergeTree::dropReplicaByZkPath

update doc and delete useless code

fix conflict

fix doc

fix doc

fix StorageReplicatedMergeTree::dropReplica

fix bug

delete useless code

* rewrite static StorageReplicatedMergeTree::dropReplica

* format comment

* SHA256 add FixedString support

* fixup

* Update index.md

* fix of incorrect example

was `curl -sS "<address>?param_id=2¶m_phrase=test" -d "SELECT * FROM table WHERE int_column = {id:UInt8} and string_column = {phrase:String}"`

* Add missed lock

* jemalloc: disable percpu_arena on non-linux (not supported anyway)

And since it does not supported it produces the following warning:
  <jemalloc>: No getcpu support: percpu_arena:percpu
  <jemalloc>: perCPU arena getcpu() not available. Setting narenas to 8.

However it is uses narenas=ncpu instead of default narenas=ncpu*4

* Optimize code

* jemalloc: disable it for darwin

Right now it aborts (Abort trap: 6) on osx, the issue does not looks
like [1], but should be investigated, so disable for now.

What is interesting that it works under debugger, so I guess it is
indeed the order of symbol replacements...

  [1]: jemalloc/jemalloc#420

* fix bug for query "select *, abc.*" should failed. (#11753)

* Remove redundat sync replica

* test commit

* Hook for push events

* add test

fix

fix

fix

* memory usage settings

* Update README.md

* Update README.md

* Merge pull request #11071 (#11874)

* Add's hasSubstr  array function

* Add's example in the english doc of hasSubStr

* Updates conditions to avoid prematurely calling isEqual

* Remove feathericons

Authored-by: Ryad ZENINE <[email protected]>

* allow implicit initialization

* Fix bad code

* Allow ClickHouse to run on Android

* Fix documentation

* Add docs about non-linux binaries (#11889)

* Add docs about non-linux binaries

* Better

* Update install.md

* Update install.md

* Update docs/en/getting-started/install.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Update docs/en/getting-started/install.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Update docs/en/getting-started/install.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Review fixes

* Better instruction

* Fix reference

* Better

* Better links

Co-authored-by: Ivan Blinkov <[email protected]>

* Fix anotehr one nullable prewhere column.

* minor improvements

* Update StorageSystemTables.cpp

* Added a test

* fix style check - removed unused LOGICAL_ERROR

* Fix if function with NULLs (#11807)

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* sync with goals

* Remove unused code

* Remove already fixed bug

* update SELECT description (#11907)

Signed-off-by: Slach <[email protected]>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Fix using current database while checking access rights.

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* for the love of God just update this container already, i'm begging you

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* Update features.html

* fixpu

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

Co-authored-by: alesapin <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Bharat Nallan <[email protected]>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: Alexey Milovidov <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: sundy-li <[email protected]>
Co-authored-by: amudong <[email protected]>
Co-authored-by: feng lv <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: Alexandr Kondratev <[email protected]>
Co-authored-by: hexiaoting <[email protected]>
Co-authored-by: Alexander Kazakov <[email protected]>
Co-authored-by: Alexander Tokmakov <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Vxider <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Eugene Klimov <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: robot-clickhouse <[email protected]>