Skip to content

Fix parquet writing not preserving original order when using single threaded writing with the native writer#90126

Merged
Avogar merged 5 commits intoClickHouse:masterfrom
arthurpassos:process_parquet_chunks_in_order
Nov 21, 2025
Merged

Fix parquet writing not preserving original order when using single threaded writing with the native writer#90126
Avogar merged 5 commits intoClickHouse:masterfrom
arthurpassos:process_parquet_chunks_in_order

Conversation

@arthurpassos
Copy link
Contributor

@arthurpassos arthurpassos commented Nov 15, 2025 •
edited
Loading

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fix parquet writing not preserving original order when using single threaded writing with the native writer. Partially reverts https://github.com/ClickHouse/ClickHouse/pull/64424/files

@Avogar Avogar self-assigned this Nov 19, 2025
@Avogar Avogar added the can be tested Allows running workflows for external contributors label Nov 19, 2025
@clickhouse-gh
Copy link
Contributor

clickhouse-gh bot commented Nov 19, 2025 •
edited
Loading

Workflow [PR], commit [b919390]

Summary: ❌

job_name test_name status info comment
Integration tests (amd_tsan, 1/6) failure
test_storage_s3_queue/test_parallel_inserts.py::test_parallel_inserts_with_failures[1] FAIL cidb, flaky
Stress test (amd_ubsan) failure
Server died FAIL cidb
Hung check failed, possible deadlock found (see hung_check.log) FAIL cidb
Killed by signal (in clickhouse-server.log) FAIL cidb
Fatal message in clickhouse-server.log (see fatal_messages.txt) FAIL cidb
Killed by signal (output files) FAIL cidb
Found signal in gdb.log FAIL cidb
BuzzHouse (amd_debug) failure
Logical error: 'Inconsistent AST formatting: the query: FAIL cidb
BuzzHouse (arm_asan) failure
Received signal 17 FAIL cidb
BuzzHouse (amd_ubsan) failure
Logical error: 'ReadBuffer is canceled. Can't read from it.'. FAIL cidb

@clickhouse-gh clickhouse-gh bot added the pr-performance Pull request with some performance improvements label Nov 19, 2025
@Avogar
Copy link
Member

Avogar commented Nov 19, 2025

Performance Improvement

Not sure abut this changelog entry. Should it be bug fix?

@arthurpassos
Copy link
Contributor Author

arthurpassos commented Nov 19, 2025

Performance Improvement

Not sure abut this changelog entry. Should it be bug fix?

I thought so too, but iirc it requires a test that validates the bug fix and I don't have the bandwidth to write one as of now

@Avogar
Copy link
Member

Avogar commented Nov 19, 2025 •
edited
Loading

I thought so too, but iirc it requires a test that validates the bug fix and I don't have the bandwidth to write one as of now

The test is easy:

$ clickhouse-local -q "select * from numbers(10) settings max_block_size=1, output_format_parquet_use_custom_encoder=1, max_threads=1 format Parquet" | clickhouse-local -q "select * from table" --input-format=Parquet
9
8
7
6
5
4
3
2
1
0

@arthurpassos
Copy link
Contributor Author

arthurpassos commented Nov 19, 2025

I thought so too, but iirc it requires a test that validates the bug fix and I don't have the bandwidth to write one as of now

The test is easy:

$ clickhouse-local -q "select * from numbers(10) settings max_block_size=1, output_format_parquet_use_custom_encoder=1, max_threads=1 format Parquet" | clickhouse-local -q "select * from table" --input-format=Parquet
9
8
7
6
5
4
3
2
1
0

Done, thanks for the test 👍

@clickhouse-gh clickhouse-gh bot added pr-bugfix Pull request with bugfix, not backported by default and removed pr-performance Pull request with some performance improvements labels Nov 19, 2025
@arthurpassos
Copy link
Contributor Author

arthurpassos commented Nov 21, 2025

@Avogar can we merge it?

@Avogar
Copy link
Member

Avogar commented Nov 21, 2025

Integration tests (amd_tsan, 1/6) - https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=90126&sha=b91939014853cbe32114d2ad319cf20d20f1464a&name_0=PR
Stress test (amd_ubsan) - #87016
BuzzHouse (amd_ubsan) - #84186
BuzzHouse (amd_debug) - #82833
BuzzHouse (arm_asan) - #88900

@Avogar Avogar added this pull request to the merge queue Nov 21, 2025
Merged via the queue into ClickHouse:master with commit c9c3ec4 Nov 21, 2025
125 of 132 checks passed
@robot-ch-test-poll robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Nov 21, 2025
@arthurpassos arthurpassos mentioned this pull request Feb 6, 2026
Merged
25 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Avogar Avogar Avogar approved these changes

Assignees

@Avogar Avogar

Labels

can be tested Allows running workflows for external contributors pr-bugfix Pull request with bugfix, not backported by default pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@arthurpassos @Avogar @robot-ch-test-poll