Changelog category (leave one):

• Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add settings output_format_orc_dictionary_key_size_threshold to allow user to enable dict encoding for string column in ORC output format. It helps reduce the output orc file size and improve reading performance significantly.

TODO

• Support EncodedStringVectorBatch for interface StringColumnWriter::add (in Improve string column dict encoding performance orc#15)

1. Write ORC file with string dict encoding enabled

set output_format_orc_dictionary_key_size_threshold = 1;
select concat('gluten ', cast(rand()%1000 as String)) from numbers(10000000) into outfile 'dict.orc' truncate;
10000000 rows in set. Elapsed: 2.794 sec. Processed 10.00 million rows, 80.00 MB (3.58 million rows/s., 28.63 MB/s.)
Peak memory usage: 9.07 MiB.

2. Write ORC file with string dict encoding disabled

set output_format_orc_dictionary_key_size_threshold = 0; 
select concat('gluten ', cast(rand()%1000 as String)) from numbers(10000000) into outfile 'no-dict.orc' truncate;
10000000 rows in set. Elapsed: 1.222 sec. Processed 10.00 million rows, 80.00 MB (8.18 million rows/s., 65.46 MB/s.)
Peak memory usage: 9.07 MiB.

3. Compare ORC file size

$ ll -rt *dict*
-rw-r--r-- 1 root root 22731823 Aug 20 15:42 no-dict.orc
-rw-r--r-- 1 root root 14921714 Aug 20 15:42 dict.orc

4. Compare reading performance of both ORC files

select * from file('dict.orc') format Null;
0 rows in set. Elapsed: 0.166 sec. Processed 10.00 million rows, 14.92 MB (60.19 million rows/s., 89.82 MB/s.)

select * from file('no-dict.orc') format Null;
0 rows in set. Elapsed: 0.385 sec. Processed 10.00 million rows, 22.73 MB (25.96 million rows/s., 59.00 MB/s.)

5. Conclusions

• Writing string column with dict encoding is 2.3x slower than without dict encoding. But I improved its performance in Improve string column dict encoding performance orc#15. Hope for your review.
• ORC files written with dict encoding is 1.5x smaller than without dict encoding
• Reading string column with dict encoding is 2.3x faster than without dict encoding

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

• Allow: All Required Checks
• Allow: Stateless tests
• Allow: Stateful tests
• Allow: Integration Tests
• Allow: Performance tests
• Allow: All Builds
• Allow: batch 1, 2 for multi-batch jobs
• Allow: batch 3, 4, 5, 6 for multi-batch jobs

• Exclude: Style check
• Exclude: Fast test
• Exclude: All with ASAN
• Exclude: All with TSAN, MSAN, UBSAN, Coverage
• Exclude: All with aarch64, release, debug

• Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
• Exclude: AST fuzzers

• Do not test
• Woolen Wolfdog
• Upload binaries for special builds
• Disable merge-commit
• Disable CI cache