Add JSON Reading/Writing Support for Parquet Files by nihalzp · Pull Request #79649 · ClickHouse/ClickHouse

nihalzp · 2025-04-27T20:03:20Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support writing CH JSON columns to Parquet and reading Parquet JSON columns directly as CH JSON columns.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Usage

Write:

CREATE TABLE example (
    id        Int32,
    json_str  String,
    json_col  JSON
) ENGINE = MergeTree
ORDER BY id;

INSERT INTO example (id, json_str, json_col) VALUES
    (1, '{"user":"alice","age":30}',         '{"user":"alice","age":30}'),
    (2, '{"items":[1,2,3],"flag":true}',     '{"items":{"yes":[1,2,3]},"flag":true}'),
    (3, '{"meta":{"x":10,"y":20}}',          null);

select * from example into outfile 'json_to_pq.parquet' TRUNCATE  format parquet;

-- json_col should have Parquet logical type JSON
select * from file('json_to_pq.parquet', ParquetMetaData);

Read:

SET input_format_parquet_enable_json_parsing = 1;

-- json_col should have CH type JSON
DESCRIBE TABLE (
SELECT *
FROM file(
    'json_to_pq.parquet',
    Parquet
));

nihalzp · 2025-04-27T20:04:30Z

Closes #78992

clickhouse-gh · 2025-04-27T22:45:16Z

Workflow [PR], commit [fa87f7b]

nihalzp · 2025-04-28T04:28:01Z

The failed tests appear to be unrelated.

src/Processors/Formats/Impl/ArrowColumnToCHColumn.cpp

src/Processors/Formats/Impl/Parquet/Write.cpp

tests/queries/0_stateless/03445_parquet_json_roundtrip.sh

Fix rebase issue: - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]>

Fix rebase issue: - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]>

Fix rebase issue: - 20250713 ClickHouse#82949 - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]>

Fix rebase issue: - 20250728 ClickHouse#84180 - 20250713 ClickHouse#82949 - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]>

Fix rebase issue: - 20250806 ClickHouse#84821 - 20250804 ClickHouse#83997 - 20250728 ClickHouse#84180 - 20250713 ClickHouse#82949 - 20250703 ClickHouse#82934 - 20250626 ClickHouse#80931 - 20250604 ClickHouse#79649 - 20250502 ClickHouse#79180 - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <[email protected]>

nihalzp added 8 commits April 27, 2025 01:34

Add JSON column writing support for Parquet

76472fa

Add JSON column parsing support for Parquet files

97d58e5

Merge branch 'master' into parquet-json-support

71c0f76

Update documentation

314ca3f

Add setting to toggle JSON parsing

05f104b

Fix bug to allow String types to be casted as JSON

e6bc835

Add tests for JSON support for parquet

314b128

Improve readability

0d0d7ea

alexey-milovidov added the can be tested Allows running workflows for external contributors label Apr 27, 2025

clickhouse-gh bot added the pr-feature Pull request with new product feature label Apr 27, 2025

Avogar self-assigned this Apr 28, 2025