Refactor how IInputFormat deals with reading many files at once by al13n321 · Pull Request #80931 · ClickHouse/ClickHouse

al13n321 · 2025-05-28T02:20:59Z

Changelog category (leave one):

Not for changelog (changelog entry is not required)

Closes #65963

Changes how e.g. SELECT ... FROM file('part{00..99}.parquet', ...) does coordination among the 100 file readers.

Goals:

Replace Utilizing a Shared Parsing Pool for Multiple Parquet Streams #66253 . I.e. if a few big files are mixed with many small files, use more threads for the big files once the small readers complete.
Give IInputFormat access to ActionsDAG of the filter condition. Previously it only had a KeyCondition constructed from that ActionsDAG. This is needed for A new parquet reader that supports filter push down, which improves the total time on clickbench by 50+% compared to arrow parquet reader #70611
Share a ThreadPoolCallbackRunnerFast instance among file readers. This is needed for [Very WIP] Yet another parquet reader #78380 . Might be useful for ParallelParsingInputFormat or ParallelParsingInputFormat if thread scheduling overhead is noticeable there; I haven't tried that.
Make it possible to share other things among the readers, e.g. memory budget or preprocessed filter expression (e.g. split by columns, to avoid repeating this work for each file).
Fix the inconsistency between max_parsing_threads and max_download_threads settings: the former is the total across all files, the latter used to be per file (i.e. total number of threads was max_download_threads * file_count). Made max_download_threads be a total as well. Not sure what to do about compatibility.

clickhouse-gh · 2025-05-28T02:21:24Z

Workflow [PR], commit [dbfe932]

Avogar

LGTM, but I see query with Parquet file in reading_from_file performance test slowed down:

Before the merge let's check if it's connected to the changes

al13n321 · 2025-06-13T22:46:42Z

I couldn't reproduce the performance difference. On my machine I'm getting ~0.33s both on master and on this PR. I guess it was a false positive?

…parquet reader v3

clickhouse-gh · 2025-06-18T23:33:02Z

Workflow [PR], commit [c789529]

Summary: ❌

job_name	test_name	status
AST fuzzer (amd_tsan)		failure
	Task failed: $?=127	FAIL
AST fuzzer (amd_ubsan)		failure
	Task failed: $?=127	FAIL
BuzzHouse (amd_tsan)		failure
	Task failed: $?=127	FAIL
BuzzHouse (amd_ubsan)		failure
	Task failed: $?=127	FAIL

…leOrThrowOnError throws

al13n321 · 2025-06-20T23:45:53Z

Stress test error looks unrelated ("Coordination error: Not authenticated"): https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=80931&sha=2ee75dc9111c991a8b099d21afec6d8af575c9e4&name_0=PR&name_1=Stress%20test%20%28amd_ubsan%29

Fix rebase issue: - 20250713 ClickHouse#82949 - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]>

Fix rebase issue: - 20250728 ClickHouse#84180 - 20250713 ClickHouse#82949 - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]>

Fix rebase issue: - 20250806 ClickHouse#84821 - 20250804 ClickHouse#83997 - 20250728 ClickHouse#84180 - 20250713 ClickHouse#82949 - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]>

Fix rebase issue: - 20250728 ClickHouse#84180 - 20250713 ClickHouse#82949 - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]>

Fix rebase issue: - 20250728 ClickHouse#84180 - 20250713 ClickHouse#82949 - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]> (cherry picked from commit c68fb33)

Fix rebase issue: - 20250728 ClickHouse#84180 - 20250713 ClickHouse#82949 - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]> (cherry picked from commit c68fb33) (cherry picked from commit 12ab7a6)

Refactor how IInputFormat deals with reading many files at once

de9fa0f

al13n321 requested a review from Avogar May 28, 2025 02:20

clickhouse-gh bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label May 28, 2025

al13n321 added 3 commits May 28, 2025 02:46

Style

742a73f

Disable clang-tidy warning that does spooky action at a distance

230d9b8

Merge remote-tracking branch 'origin/master' into shp

08f921f

al13n321 marked this pull request as ready for review May 28, 2025 19:34

pamarcos assigned Avogar May 29, 2025

al13n321 mentioned this pull request May 29, 2025

A new parquet reader that supports filter push down, which improves the total time on clickbench by 50+% compared to arrow parquet reader #70611

Closed

21 tasks

Avogar reviewed Jun 5, 2025

View reviewed changes

al13n321 added 2 commits June 13, 2025 21:28

Merge remote-tracking branch 'origin/master' into shp

9d72413

Use shared pool in ParallelParsingInputFormat and DWARFBlockInputFormat

7ce16d2

al13n321 added 2 commits June 13, 2025 22:54

Use different metric names for parquet thread pools, to match future …

7d5a0a8

…parquet reader v3

Use shared ThreadPoolCallbackRunnerFast in ParquetBlockInputFormat

dbfe932

Avogar approved these changes Jun 18, 2025

View reviewed changes

al13n321 added 2 commits June 18, 2025 20:51

Initialize getFormatParsingThreadPool in ClientBase

4d7e30f

Merge remote-tracking branch 'origin/master' into shp

5efb430

al13n321 and others added 9 commits June 19, 2025 01:39

Initialize getFormatParsingThreadPool in ClientBase some more

67b0fa3

Where did this come from

5e68941

Perhaps I should compile the code before pushing it

130eacb

More initFormatParsingThreadPoolIfNeeded()

05ecbe1

Lazily init shared thread pool in FormatFactory instead

ff60131

Merge branch 'master' into shp

84237a2

Fix broken build

635b0ae

Merge remote-tracking branch 'origin/master' into shp

430711c

Fix build, fix ThreadPoolCallbackRunnerFast getting stuck when schedu…

2ee75dc

…leOrThrowOnError throws

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor how IInputFormat deals with reading many files at once#80931

Refactor how IInputFormat deals with reading many files at once#80931
al13n321 merged 22 commits intomasterfrom
shp

al13n321 commented May 28, 2025

Uh oh!

clickhouse-gh bot commented May 28, 2025 •

edited

Loading

Uh oh!

Avogar left a comment •

edited

Loading

Uh oh!

al13n321 commented Jun 13, 2025

Uh oh!

clickhouse-gh bot commented Jun 18, 2025 •

edited

Loading

Uh oh!

al13n321 commented Jun 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

al13n321 commented May 28, 2025

Changelog category (leave one):

Uh oh!

clickhouse-gh bot commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Avogar left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

al13n321 commented Jun 13, 2025

Uh oh!

clickhouse-gh bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

al13n321 commented Jun 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

clickhouse-gh bot commented May 28, 2025 •

edited

Loading

Avogar left a comment •

edited

Loading

clickhouse-gh bot commented Jun 18, 2025 •

edited

Loading