Skip to content

Allow reading LowCardinality(String/FixedString) data from orc dictionary encoded columns#69481

Merged
alexey-milovidov merged 25 commits intoClickHouse:masterfrom
bigo-sg:poc_orc_lc_read
Oct 15, 2024
Merged

Allow reading LowCardinality(String/FixedString) data from orc dictionary encoded columns#69481
alexey-milovidov merged 25 commits intoClickHouse:masterfrom
bigo-sg:poc_orc_lc_read

Conversation

@taiyang-li
Copy link
Contributor

@taiyang-li taiyang-li commented Sep 11, 2024

Changelog category (leave one):

  • Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Enable reading LowCardinality string columns from ORC.

This PR must be reviewed after #68591

1 Construct the data

set output_format_orc_dictionary_key_size_threshold = 1;
select concat('gluten ', cast(rand()%1000 as String)) from numbers(10000000) into outfile 'dict.orc' truncate;

2 Readind dict.orc and execute group by (speedup 2.6x)

set input_format_orc_dictionary_as_low_cardinality = 0; 
select "concat('gluten ', CAST(modulo(rand(), 1000), 'String'))" as x, count(1) from  file('dict.orc') group by x format Null; 
0 rows in set. Elapsed: 0.485 sec. Processed 10.00 million rows, 14.92 MB (20.63 million rows/s., 30.79 MB/s.)

set input_format_orc_dictionary_as_low_cardinality = 1; 
select "concat('gluten ', CAST(modulo(rand(), 1000), 'String'))" as x, count(1) from  file('dict.orc') group by x format Null;
0 rows in set. Elapsed: 0.190 sec. Processed 10.00 million rows, 14.92 MB (52.57 million rows/s., 78.43 MB/s.)

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

  • Allow: All Required Checks
  • Allow: Stateless tests
  • Allow: Stateful tests
  • Allow: Integration Tests
  • Allow: Performance tests
  • Allow: All Builds
  • Allow: batch 1, 2 for multi-batch jobs
  • Allow: batch 3, 4, 5, 6 for multi-batch jobs

  • Exclude: Style check
  • Exclude: Fast test
  • Exclude: All with ASAN
  • Exclude: All with TSAN, MSAN, UBSAN, Coverage
  • Exclude: All with aarch64, release, debug

  • Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
  • Exclude: AST fuzzers

  • Do not test
  • Woolen Wolfdog
  • Upload binaries for special builds
  • Disable merge-commit
  • Disable CI cache

@robot-ch-test-poll robot-ch-test-poll added the pr-performance Pull request with some performance improvements label Sep 11, 2024
@robot-ch-test-poll3
Copy link
Contributor

robot-ch-test-poll3 commented Sep 11, 2024

This is an automated comment for commit 8cbaec9 with description of existing statuses. It's updated for the latest CI running

✅ Click here to open a full report in a separate page

Successful checks
Check nameDescriptionStatus
BuildsThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
Docs checkBuilds and tests the documentation✅ success
Fast testNormally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here✅ success
Flaky testsChecks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc✅ success
Integration testsThe integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests✅ success
Performance ComparisonMeasure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests✅ success
Stateful testsRuns stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Stateless testsRuns stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success
Unit testsRuns the unit tests for different release types✅ success

@taiyang-li taiyang-li changed the title [WIP] Enable reading lowcardinality string columns from ORC [WIP] Enable reading lowcardinality string columns from orc dictionary encoded columns Sep 11, 2024
@taiyang-li taiyang-li changed the title [WIP] Enable reading lowcardinality string columns from orc dictionary encoded columns [WIP] Enable reading LowCardinality(String/FixedString) data from orc dictionary encoded columns Sep 11, 2024
@taiyang-li taiyang-li changed the title [WIP] Enable reading LowCardinality(String/FixedString) data from orc dictionary encoded columns [WIP] Allow reading LowCardinality(String/FixedString) data from orc dictionary encoded columns Sep 11, 2024
@taiyang-li taiyang-li changed the title [WIP] Allow reading LowCardinality(String/FixedString) data from orc dictionary encoded columns Allow reading LowCardinality(String/FixedString) data from orc dictionary encoded columns Sep 11, 2024
@taiyang-li taiyang-li marked this pull request as ready for review September 11, 2024 12:27
@taiyang-li taiyang-li marked this pull request as draft September 11, 2024 12:30
@taiyang-li taiyang-li marked this pull request as ready for review September 12, 2024 02:11
@taiyang-li
Copy link
Contributor Author

@nickitat hope for your reviews, thanks!

Copy link
Member

@alexey-milovidov alexey-milovidov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are missing.

@alexey-milovidov alexey-milovidov self-assigned this Sep 21, 2024
@alexey-milovidov alexey-milovidov added this pull request to the merge queue Oct 15, 2024
Merged via the queue into ClickHouse:master with commit dfbd356 Oct 15, 2024
@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Oct 15, 2024
baibaichen added a commit to Kyligence/gluten that referenced this pull request Oct 16, 2024
baibaichen added a commit to Kyligence/gluten that referenced this pull request Oct 16, 2024
baibaichen added a commit to apache/gluten that referenced this pull request Oct 16, 2024
* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20241016)

* Fix Build due to ClickHouse/ClickHouse#70158

* Fix build due to ClickHouse/ClickHouse#62966

* add more info for exception

* Fix ut due to ClickHouse/ClickHouse#69481

---------

Co-authored-by: kyligence-git <[email protected]>
Co-authored-by: Chang Chen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-performance Pull request with some performance improvements pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants