Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes by filimonov · Pull Request #79969 · ClickHouse/ClickHouse

filimonov · 2025-05-08T13:25:38Z

Changelog category (leave one):

Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed incorrect query results and out-of-memory crashes when using match(column, '^…') with backslash-escaped characters.

Documentation entry for user-facing changes

Previously, hitting an unsupported “\x” escape in ^… patterns would do pos = end but still run a stray ++pos, so the code walked past the regex buffer, pulled in garbage bytes (the “�i” etc), and could OOM.

Now that stray increment is removed and all single-char escapes are handled correctly.

Backports are needed.

See example of the problem in fiddle: https://fiddle.clickhouse.com/ef60eb3a-8613-4127-b5b4-bb62fc274748

Another symptom is a very quick memory allocations (up to OOM) with a traces like below:

std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::push_back(char)
DB::KeyCondition::$_20::__invoke(DB::KeyCondition::RPNElement&, DB::Field const&)
DB::KeyCondition::extractAtomFromTree(DB::RPNBuilderTreeNode const&, DB::KeyCondition::RPNElement&)
DB::RPNBuilder<DB::KeyCondition::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&)
DB::RPNBuilder<DB::KeyCondition::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&)
DB::KeyCondition::KeyCondition(DB::ActionsDAG const*, std::__1::shared_ptr<DB::Context const>, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::shared_ptr<DB::ExpressionActions> const&, bool)
DB::buildIndexes(std::__1::optional<DB::ReadFromMergeTree::Indexes>&, DB::ActionsDAG const*, DB::MergeTreeData const&, std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeDataPart const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeDataPart const>>> const&, std::__1::shared_ptr<DB::MergeTreeData::IMutationsSnapshot const> const&, std::__1::shared_ptr<DB::Context const> const&, DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::shared_ptr<Poco::Logger> const&)
DB::ReadFromMergeTree::applyFilters(DB::ActionDAGNodes)
DB::QueryPlanOptimizations::optimizePrimaryKeyConditionAndLimit(std::__1::vector<DB::QueryPlanOptimizations::Frame, std::__1::allocator<DB::QueryPlanOptimizations::Frame>> const&)
DB::QueryPlanOptimizations::optimizeTreeSecondPass(DB::QueryPlanOptimizationSettings const&, DB::QueryPlan::Node&, std::__1::list<DB::QueryPlan::Node, std::__1::allocator<DB::QueryPlan::Node>>&)
DB::QueryPlan::optimize(DB::QueryPlanOptimizationSettings const&)
DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&)
DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline()
DB::InterpreterSelectQueryAnalyzer::execute()
DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*)
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum)
DB::TCPHandler::runImpl()
DB::TCPHandler::run()
Poco::Net::TCPServerConnection::start()
Poco::Net::TCPServerDispatcher::run()
Poco::PooledThread::run()
Poco::ThreadImpl::runnableEntry(void*)

- Change loop to `while (pos < end)` to guard against overruns - Support all single‐char escapes: `\]`, `\}`, `\-`, `\\` alongside existing metachars - Move `++pos` into the supported‐escape case only, removing stray increments - Prevent garbage reads past `end` and ensure correct fixed‐prefix for escaped metacharacters

filimonov · 2025-05-08T13:49:38Z

src/Storages/MergeTree/KeyCondition.cpp

                    default:
                        /// all other escape sequences are not supported
                            pos = end;
-                    break;


after that break (which exits the current switch) the increment on line 151 was occurring, moving the pos pointer after end of string.

clickhouse-gh · 2025-05-08T20:39:21Z

Workflow [PR], commit [d483324]

filimonov · 2025-05-09T18:06:59Z

Bugfix check fails because of
Unknown setting 'allow_dynamic_cache_resize'

Indeed that setting was missing in 25.4...

GrigoryPervakov

LGTM, but check your test failure. Probably some random settings broke it and it needs no-random-settings to be stable in ci

tests/queries/0_stateless/03456_match_index_prefix_extraction.sql

Co-authored-by: Pervakov Grigorii <[email protected]>

GrigoryPervakov · 2025-06-06T14:47:33Z

Fuzzer error is not related, it reproduces in master #81431

Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes

24.8.14 Backport of ClickHouse#79969 -- Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes

filimonov · 2025-10-31T13:40:27Z

Backports? It really can lead to crashes.

Cherry pick #79969 to 25.3: Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes

…aped metacharacters to avoid wrong results and crashes

Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes

25.3.8 Backport of ClickHouse#79969 - Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes

filimonov commented May 8, 2025

View reviewed changes

alexey-milovidov added the can be tested Allows running workflows for external contributors label May 8, 2025

clickhouse-gh bot added the pr-bugfix Pull request with bugfix, not backported by default label May 8, 2025

filimonov added 2 commits May 9, 2025 12:30

Rerun ci/cd

8487dc6

Merge remote-tracking branch 'upstream/master' into fix_index_match

f09b131

GrigoryPervakov self-assigned this May 12, 2025

GrigoryPervakov approved these changes May 12, 2025

View reviewed changes

Merge branch 'master' into fix_index_match

ba67f71

GrigoryPervakov reviewed Jun 4, 2025

View reviewed changes

tests/queries/0_stateless/03456_match_index_prefix_extraction.sql Show resolved Hide resolved

Update tests/queries/0_stateless/03456_match_index_prefix_extraction.sql

d483324

Co-authored-by: Pervakov Grigorii <[email protected]>

GrigoryPervakov added this pull request to the merge queue Jun 6, 2025

Merged via the queue into ClickHouse:master with commit 79902f2 Jun 6, 2025
117 of 121 checks passed

robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Jun 6, 2025

zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Jun 17, 2025

Merge pull request ClickHouse#79969 from filimonov/fix_index_match

3a11a83

Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes

zvonand mentioned this pull request Jun 17, 2025

24.8.14 Backport of #79969 -- Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes Altinity/ClickHouse#863

Merged

zvonand mentioned this pull request Jun 23, 2025

24.8.14 Pre-release PR Altinity/ClickHouse#827

Merged

KochetovNicolai added pr-must-backport Pull request should be backported intentionally. Use this label with great care! pr-must-backport-cloud labels Oct 31, 2025

robot-clickhouse-ci-2 added pr-backports-created-cloud deprecated label, NOOP pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR labels Oct 31, 2025

robot-ch-test-poll3 mentioned this pull request Oct 31, 2025

Cherry pick #79969 to 25.3: Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes #89328

Merged

robot-ch-test-poll3 added a commit that referenced this pull request Oct 31, 2025

Merge pull request #89328 from ClickHouse/cherrypick/25.3/79969

09185ad

Cherry pick #79969 to 25.3: Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes

robot-clickhouse added a commit that referenced this pull request Oct 31, 2025

Backport #79969 to 25.3: Fix match(col, '^…') index analysis with esc…

1fdc3ea

…aped metacharacters to avoid wrong results and crashes

robot-ch-test-poll3 mentioned this pull request Oct 31, 2025

Backport #79969 to 25.3: Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes #89331

Open

robot-clickhouse-ci-1 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Oct 31, 2025

mkmkme pushed a commit to Altinity/ClickHouse that referenced this pull request Nov 4, 2025

Merge pull request ClickHouse#79969 from filimonov/fix_index_match

922bc38

Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes

mkmkme mentioned this pull request Nov 4, 2025

25.3.8 Backport of #79969 - Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes Altinity/ClickHouse#1122

Merged

25 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes#79969

Fix match(col, '^…') index analysis with escaped metacharacters to avoid wrong results and crashes#79969
GrigoryPervakov merged 5 commits intoClickHouse:masterfrom
filimonov:fix_index_match

filimonov commented May 8, 2025 •

edited

Loading

Uh oh!

filimonov May 8, 2025 •

edited

Loading

Uh oh!

clickhouse-gh bot commented May 8, 2025 •

edited

Loading

Uh oh!

filimonov commented May 9, 2025

Uh oh!

GrigoryPervakov left a comment

Uh oh!

Uh oh!

GrigoryPervakov commented Jun 6, 2025

Uh oh!

Uh oh!

filimonov commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

filimonov commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

filimonov May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clickhouse-gh bot commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filimonov commented May 9, 2025

Uh oh!

GrigoryPervakov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

GrigoryPervakov commented Jun 6, 2025

Uh oh!

Uh oh!

filimonov commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

filimonov commented May 8, 2025 •

edited

Loading

filimonov May 8, 2025 •

edited

Loading

clickhouse-gh bot commented May 8, 2025 •

edited

Loading