Skip to content

Crash in IN function where columns have different types and many columns are involved#89367

Merged
KochetovNicolai merged 7 commits intoClickHouse:masterfrom
ilejn:in_function_crash
Nov 13, 2025
Merged

Crash in IN function where columns have different types and many columns are involved#89367
KochetovNicolai merged 7 commits intoClickHouse:masterfrom
ilejn:in_function_crash

Conversation

@ilejn
Copy link
Contributor

@ilejn ilejn commented Nov 1, 2025

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Possible crash/undefined behavior in IN function where primary key column types are different from IN function right side column types. Example: SELECT string_column, int_column FROM test_table WHERE (string_column, int_column) IN (SELECT '5', 'not a number'). Appears if many rows are selected and there are rows contain not compatible types.

@ilejn
Copy link
Contributor Author

ilejn commented Nov 3, 2025

Query

WITH latest AS ( SELECT Account, Symbol, 
 timestamp FROM test.t1
 ), 
source AS ( SELECT * FROM test.t2
 WHERE ((Symbol, Account, timestamp) IN (latest))
SELECT * FROM source;

The thing about the query - match between (Account, Symbol) and (Symbol, Account) which is UInt32 against String. Some (not all) Symbol values can be casted to UInt32.

Backtrace

2025.10.27 14:52:40.024878 [ 115459 ] {} <Fatal> ClientBase: 3. __pthread_kill @ 0x000000000009eb2d
2025.10.27 14:52:40.024905 [ 115459 ] {} <Fatal> ClientBase: 4. gsignal @ 0x000000000004527e
2025.10.27 14:52:40.024918 [ 115459 ] {} <Fatal> ClientBase: 5. pzero @ 0x00000000000288ff
2025.10.27 14:52:40.024946 [ 115459 ] {} <Fatal> ClientBase: 6. _nl_load_domain.cold @ 0x000000000002881b
2025.10.27 14:52:40.024977 [ 115459 ] {} <Fatal> ClientBase: 7. ? @ 0x000000000003b517
2025.10.27 14:52:40.537002 [ 115459 ] {} <Fatal> ClientBase: 8.0. inlined from /home/ilejn/projects/ClickHouse/src/Common/PODArray.h:407: DB::PODArray<unsigned int, 4096ul, Allocator<false, false>, 63ul, 64ul>::operator[](long) const
2025.10.27 14:52:40.537071 [ 115459 ] {} <Fatal> ClientBase: 8.1. inlined from /home/ilejn/projects/ClickHouse/src/Columns/ColumnVector.cpp:109: DB::ColumnVector<unsigned int>::less::operator()(unsigned long, unsigned long) const
2025.10.27 14:52:40.537095 [ 115459 ] {} <Fatal> ClientBase: 8. /home/ilejn/projects/ClickHouse/base/base/../base/sort.h:0: bool DebugLessComparator<DB::ColumnVector<unsigned int>::less>::operator()<unsigned long, unsigned long>(unsigned long&, unsigned long&) @ 0x0000000019453df0
2025.10.27 14:52:40.881700 [ 115459 ] {} <Fatal> ClientBase: 9.0. inlined from /home/ilejn/projects/ClickHouse/contrib/pdqsort/pdqsort.h:154: void pdqsort_detail::sort2<unsigned long*, DebugLessComparator<DB::ColumnVector<unsigned int>::less>>(unsigned long*, unsigned long*, DebugLessComparator<DB::ColumnVector<unsigned int>::less>)
2025.10.27 14:52:40.881769 [ 115459 ] {} <Fatal> ClientBase: 9.1. inlined from /home/ilejn/projects/ClickHouse/contrib/pdqsort/pdqsort.h:160: void pdqsort_detail::sort3<unsigned long*, DebugLessComparator<DB::ColumnVector<unsigned int>::less>>(unsigned long*, unsigned long*, unsigned long*, DebugLessComparator<DB::ColumnVector<unsigned int>::less>)
2025.10.27 14:52:40.881788 [ 115459 ] {} <Fatal> ClientBase: 9. /home/ilejn/projects/ClickHouse/contrib/pdqsort/pdqsort.h:436: void pdqsort_detail::pdqsort_loop<unsigned long*, DebugLessComparator<DB::ColumnVector<unsigned int>::less>, false>(unsigned long*, unsigned long*, DebugLessComparator<DB::ColumnVector<unsigned int>::less>, int, bool) @ 0x0000000019459e47
2025.10.27 14:52:41.103444 [ 115459 ] {} <Fatal> ClientBase: 10.0. inlined from /home/ilejn/projects/ClickHouse/contrib/pdqsort/pdqsort.h:675: void pdqsort<unsigned long*, DebugLessComparator<DB::ColumnVector<unsigned int>::less>>(unsigned long*, unsigned long*, DebugLessComparator<DB::ColumnVector<unsigned int>::less>)
2025.10.27 14:52:41.103504 [ 115459 ] {} <Fatal> ClientBase: 10.1. inlined from /home/ilejn/projects/ClickHouse/base/base/../base/sort.h:128: void sort<unsigned long*, DB::ColumnVector<unsigned int>::less>(unsigned long*, unsigned long*, DB::ColumnVector<unsigned int>::less)
2025.10.27 14:52:41.103533 [ 115459 ] {} <Fatal> ClientBase: 10. /home/ilejn/projects/ClickHouse/src/Columns/ColumnVector.cpp:374: auto DB::ColumnVector<unsigned int>::updatePermutation(DB::IColumn::PermutationSortDirection, DB::IColumn::PermutationSortStability, unsigned long, int, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>&, std::vector<DB::EqualRange, std::allocator<DB::EqualRange>>&) const::'lambda'(auto, auto, auto)::operator()<unsigned long*, unsigned long*, DB::ColumnVector<unsigned int>::less>(auto, auto, auto) const @ 0x0000000019460391
2025.10.27 14:52:41.342111 [ 115459 ] {} <Fatal> ClientBase: 11. /home/ilejn/projects/ClickHouse/src/Columns/IColumnImpl.h:154: void DB::IColumn::updatePermutationImpl<DB::ColumnVector<unsigned int>::less, DB::ColumnVector<unsigned int>::equals, DB::ColumnVector<unsigned int>::updatePermutation(DB::IColumn::PermutationSortDirection, DB::IColumn::PermutationSortStability, unsigned long, int, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>&, std::vector<DB::EqualRange, std::allocator<DB::EqualRange>>&) const::'lambda'(auto, auto, auto), DB::ColumnVector<unsigned int>::updatePermutation(DB::IColumn::PermutationSortDirection, DB::IColumn::PermutationSortStability, unsigned long, int, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>&, std::vector<DB::EqualRange, std::allocator<DB::EqualRange>>&) const::'lambda'(auto, auto, auto, auto)>(unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>&, std::vector<DB::EqualRange, std::allocator<DB::EqualRange>>&, auto, auto, auto, auto) const @ 0x00000000193c5b88
2025.10.27 14:52:41.557368 [ 115459 ] {} <Fatal> ClientBase: 12. /home/ilejn/projects/ClickHouse/src/Columns/ColumnVector.cpp:380: DB::ColumnVector<unsigned int>::updatePermutation(DB::IColumn::PermutationSortDirection, DB::IColumn::PermutationSortStability, unsigned long, int, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>&, std::vector<DB::EqualRange, std::allocator<DB::EqualRange>>&) const @ 0x00000000193c5950
2025.10.27 14:52:41.618749 [ 115459 ] {} <Fatal> ClientBase: 13. /home/ilejn/projects/ClickHouse/src/Interpreters/sortBlock.cpp:195: DB::(anonymous namespace)::getBlockSortPermutationImpl(DB::Block const&, DB::SortDescription const&, DB::IColumn::PermutationSortStability, unsigned long, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>&) @ 0x0000000016d2ce6b
2025.10.27 14:52:41.663688 [ 115459 ] {} <Fatal> ClientBase: 14. /home/ilejn/projects/ClickHouse/src/Interpreters/sortBlock.cpp:339: DB::sortBlock(DB::Block&, DB::SortDescription const&, unsigned long, DB::IColumn::PermutationSortStability) @ 0x0000000016d2c7c7
2025.10.27 14:52:41.811810 [ 115459 ] {} <Fatal> ClientBase: 15. /home/ilejn/projects/ClickHouse/src/Interpreters/Set.cpp:658: DB::MergeTreeSetIndex::MergeTreeSetIndex(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&, std::vector<DB::MergeTreeSetIndex::KeyTuplePositionMapping, std::allocator<DB::MergeTreeSetIndex::KeyTuplePositionMapping>>&&) @ 0x0000000016a437c0

The version is close to recent master.

@ilejn
Copy link
Contributor Author

ilejn commented Nov 4, 2025

I have a repro, but it is not a small one, so I don't think that it make sense to create a test based on it.

@ilejn ilejn marked this pull request as ready for review November 4, 2025 09:53
@ilejn ilejn changed the title in_function_crash: initial Crash in IN function where columns have different types and many columns are imvolved Nov 4, 2025
@ilejn ilejn changed the title Crash in IN function where columns have different types and many columns are imvolved Crash in IN function where columns have different types and many columns are involved Nov 4, 2025
@ilejn
Copy link
Contributor Author

ilejn commented Nov 5, 2025

Hello,
could someone please enable tests for the PR and may be review it.

@ilejn
Copy link
Contributor Author

ilejn commented Nov 6, 2025

Hello @KochetovNicolai ,
sorry to bother,
could you have a look please (since you had reviewed #54544 ).

@serxa serxa added the can be tested Allows running workflows for external contributors label Nov 7, 2025
@clickhouse-gh
Copy link
Contributor

clickhouse-gh bot commented Nov 7, 2025

Workflow [PR], commit [5d4d21b]

Summary:

job_name test_name status info comment
Bugfix validation (functional tests) failure
03635_in_function_different_types_many_columns FAIL cidb
BuzzHouse (amd_debug) failure
Buzzing result failure cidb

@clickhouse-gh clickhouse-gh bot added the pr-bugfix Pull request with bugfix, not backported by default label Nov 7, 2025
{
/// We cannot afford filtering rows because
/// sortBlock relies on equal number of rows in all columns
transformed_set_columns[set_element_index] = std::move(nullable_set_column);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not look correct to me. I think the code is written in a way that tuple columns and PK columns must have the same type, and we have to apply a cast in one or another way.

The proper solution seems to be creating a common nullmask. Because if one tuple component can't be converted to desired PK type, the whole tuple can't match.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @KochetovNicolai , thank you for looking into this.

What we are doing here is exactly applying cast

         ColumnPtr nullable_set_column = castColumnAccurateOrNull({set_column, set_element_type, {}}, key_column_type);
         ...
         if (set_columns.size() > 1)
         {
             transformed_set_columns[set_element_index] = std::move(nullable_set_column);
             continue;

What we are not doing is applying filter.

Does it make sense?

Regarding 'creating a common nullmask' , I agree, it looks like a mature approach. I hesitated to go this way, because it makes the fix a bit more complex.
If you like, I can do this in another PR or switch to it in this PR.

Copy link
Member

@KochetovNicolai KochetovNicolai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option is remove column from condition completely, if there are few and at least some values don't cast.

But common filter mask looks better.

@ilejn
Copy link
Contributor Author

ilejn commented Nov 11, 2025

Hello @KochetovNicolai ,
I've implemented 'proper solution seems to be creating a common nullmask. Because if one tuple component can't be converted to desired PK type, the whole tuple can't match' as I take it.
Could you review please?

One of the test failures is caused by this PR, it must be listed in parallel_replicas_blacklist.txt . Others seem not related.

@KochetovNicolai KochetovNicolai added this pull request to the merge queue Nov 13, 2025
@ilejn
Copy link
Contributor Author

ilejn commented Nov 13, 2025

Merged via the queue into ClickHouse:master with commit 33a028c Nov 13, 2025
251 of 256 checks passed
@robot-clickhouse robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Nov 13, 2025
ilejn pushed a commit to Altinity/ClickHouse that referenced this pull request Nov 19, 2025
Crash in IN function where columns have different types and many columns are involved
zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Dec 8, 2025
Crash in IN function where columns have different types and many columns are involved
zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Dec 15, 2025
Crash in IN function where columns have different types and many columns are involved
zvonand added a commit to Altinity/ClickHouse that referenced this pull request Dec 17, 2025
25.8.12 Backport of ClickHouse#89367: Crash in IN function where columns have different types and many columns are involved
zvonand added a commit to Altinity/ClickHouse that referenced this pull request Dec 23, 2025
25.8.12 Backport of ClickHouse#89367: Crash in IN function where columns have different types and many columns are involved
zvonand added a commit to Altinity/ClickHouse that referenced this pull request Dec 25, 2025
25.8.13 Backport of ClickHouse#89367: Crash in IN function where columns have different types and many columns are involved
zvonand added a commit to Altinity/ClickHouse that referenced this pull request Jan 15, 2026
25.8.13 Backport of ClickHouse#89367: Crash in IN function where columns have different types and many columns are involved
zvonand added a commit to Altinity/ClickHouse that referenced this pull request Jan 27, 2026
25.8.13 Backport of ClickHouse#89367: Crash in IN function where columns have different types and many columns are involved
zvonand added a commit to Altinity/ClickHouse that referenced this pull request Jan 28, 2026
25.8.15 Backport of ClickHouse#89367: Crash in IN function where columns have different types and many columns are involved
zvonand added a commit to Altinity/ClickHouse that referenced this pull request Feb 9, 2026
…in_in_function

24.8.14 Backport of ClickHouse#89367 - Crash in IN function where columns have different types and many columns are involved
zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Feb 18, 2026
Crash in IN function where columns have different types and many columns are involved
zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Mar 17, 2026
Crash in IN function where columns have different types and many columns are involved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors pr-bugfix Pull request with bugfix, not backported by default pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants