Skip to content

Support parquet integer logical types on native reader#72105

Merged
alexey-milovidov merged 7 commits intoClickHouse:masterfrom
arthurpassos:parquet_native_reader_int_logical
Jan 9, 2025
Merged

Support parquet integer logical types on native reader#72105
alexey-milovidov merged 7 commits intoClickHouse:masterfrom
arthurpassos:parquet_native_reader_int_logical

Conversation

@arthurpassos
Copy link
Contributor

@arthurpassos arthurpassos commented Nov 19, 2024

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support parquet integer logical types on native reader

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

  • Allow: All Required Checks
  • Allow: Stateless tests
  • Allow: Stateful tests
  • Allow: Integration Tests
  • Allow: Performance tests
  • Allow: All Builds
  • Allow: batch 1, 2 for multi-batch jobs
  • Allow: batch 3, 4, 5, 6 for multi-batch jobs
  • Exclude: Style check
  • Exclude: Fast test
  • Exclude: All with ASAN
  • Exclude: All with TSAN, MSAN, UBSAN, Coverage
  • Exclude: All with aarch64, release, debug
  • Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
  • Exclude: AST fuzzers
  • Do not test
  • Woolen Wolfdog
  • Upload binaries for special builds
  • Disable merge-commit
  • Disable CI cache

@arthurpassos arthurpassos force-pushed the parquet_native_reader_int_logical branch from 5e8a5a6 to 92a1f0c Compare November 20, 2024 13:18
arthurpassos
arthurpassos commented Nov 20, 2024
View reviewed changes
src/Processors/Formats/Impl/Parquet/ParquetLeafColReader.cpp
template class ParquetLeafColReader<ColumnUInt32>;
template class ParquetLeafColReader<ColumnInt64>;
template class ParquetLeafColReader<ColumnUInt64>;
template class ParquetLeafColReader<ColumnBFloat16>;
Copy link
Contributor Author

@arthurpassos arthurpassos Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexey-milovidov did you add it just for the sake of adding it? As far as I can tell, this line has no effect since makeLeafReader is never called with ColumnBFloat16.

Copy link
Member

@alexey-milovidov alexey-milovidov Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea about this code.

Copy link
Contributor Author

@arthurpassos arthurpassos Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Git blame on https://github.com/ClickHouse/ClickHouse/blame/master/src/Processors/Formats/Impl/Parquet/ParquetLeafColReader.cpp says you authored it

@arthurpassos
Copy link
Contributor Author

arthurpassos commented Nov 20, 2024

Reading integer logical types is easy, the values are physically stored using the physical type with sign extension. One just has to interpret the values correctly based on logical type size.

Then why such a big PR?

As of now, ParquetLeafColReader instantiates a ParquetDataReader templated on ClickHouse column type. This is a limitation because the decision of how many bytes to read can't be made only on the ClickHouse column type. For instance: parquet boolean and logical type uint8_t will both land on the same ClickHouse column type (uint8_t), but their physical representation is different. Former is bit based, latter is byte based.

One might argue ParquetDataReader does have a way to dictacte the logical type and infer how many bytes and how to read the data from the buffer based on parquet::ColumnDescriptor. Yes, this is true. But the check for physical and logical types already happens in ParquetRecorReader, thus this check would have to be duplicated.

Hence, the refactoring. I don't have a strong case for this design, tho. Suggestions would be great.

@vdimir vdimir added the can be tested Allows running workflows for external contributors label Nov 20, 2024
@robot-clickhouse robot-clickhouse added the pr-improvement Pull request with some product improvements label Nov 20, 2024
@robot-ch-test-poll2
Copy link
Contributor

robot-ch-test-poll2 commented Nov 20, 2024 •
edited by robot-ch-test-poll1
Loading

This is an automated comment for commit e8036a0 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check nameDescriptionStatus
Integration testsThe integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests❌ failure
Successful checks
Check nameDescriptionStatus
BuildsThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Docs checkBuilds and tests the documentation✅ success
Fast testNormally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here✅ success
Flaky testsChecks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc✅ success
Stateful testsRuns stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Stateless testsRuns stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success
Unit testsRuns the unit tests for different release types✅ success

@arthurpassos arthurpassos marked this pull request as ready for review November 20, 2024 19:54
@arthurpassos arthurpassos mentioned this pull request Nov 20, 2024
Closed
@alexey-milovidov
Copy link
Member

alexey-milovidov commented Dec 28, 2024

@arthurpassos, tests have failed.

@alexey-milovidov alexey-milovidov added the close in a month if not active This will be closed in case of no information label Dec 28, 2024
@arthurpassos
Copy link
Contributor Author

arthurpassos commented Dec 29, 2024

@arthurpassos, tests have failed.

#72922.

I'll merge it with master

@arthurpassos
Copy link
Contributor Author

arthurpassos commented Jan 9, 2025

@alexey-milovidov can I get a review?

test_storage_kafka - #68360 and #68241

@alexey-milovidov alexey-milovidov removed the close in a month if not active This will be closed in case of no information label Jan 9, 2025
@alexey-milovidov alexey-milovidov self-assigned this Jan 9, 2025
@alexey-milovidov alexey-milovidov merged commit 7c327b7 into ClickHouse:master Jan 9, 2025
@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jan 9, 2025
This was referenced Jan 9, 2025
Closed
Merged
@nickitat nickitat mentioned this pull request Jan 13, 2025
Closed
@al13n321
Copy link
Member

al13n321 commented Feb 14, 2025

Is this still relevant or should we wait for #70611 instead?

@arthurpassos
Copy link
Contributor Author

arthurpassos commented Feb 14, 2025

Is this still relevant or should we wait for #70611 instead?

This PR is already merged, what do you mean by "is this still relevant"?

@al13n321
Copy link
Member

al13n321 commented Feb 14, 2025

Oops, nevermind.

(Github sometimes updates the information on a web page without refreshing and sometimes doesn't. This time it didn't, and I didn't notice that the PR is merged.)

@arthurpassos arthurpassos mentioned this pull request Mar 10, 2025
Closed
@arthurpassos arthurpassos mentioned this pull request Mar 10, 2025
Merged
1 task
@svb-alt svb-alt mentioned this pull request May 23, 2025
Closed
31 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexey-milovidov alexey-milovidov alexey-milovidov left review comments

Assignees

@alexey-milovidov alexey-milovidov

Labels

can be tested Allows running workflows for external contributors pr-improvement Pull request with some product improvements pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@arthurpassos @robot-ch-test-poll2 @alexey-milovidov @al13n321 @vdimir @robot-clickhouse @robot-ch-test-poll1