Skip to content

Can not import a DateTime64(9) column #38

@aadant

Description

@aadant

Repro : creating a Parquet file with a DateTime64(9) column :

clickhouse-local --query "select toDateTime64(now(9),9) as current_time format Parquet" > test.parquet
./ice insert test.test -p file://$HOME/test.parquet
2025-06-25 11:10:01 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: processing
2025-06-25 11:10:01 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: jvm: heap: init=3008M,used=16M,committed=224M,free=143599M,max=143616M, gc pressure: 100.00%(12ms), off-heap: init=7M,used=52M,committed=54M, open fds: 11
2025-06-25 11:10:01 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: copying to s3://warehouse/gw_logs/test3/data/1750867801084-06a056686f5853ecfc8ab14ec56fc10fee707ecbe9b929a3b53236991e9fee4f.parquet
2025-06-25 11:10:01 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: adding data file (copy took 0s)
2025-06-25 11:10:02 [main] INFO o.a.i.SnapshotProducer > Committed snapshot 3966130372501728897 (MergeAppend)
(.venv) [aadant@fpif1-dbal2 examples]$ ./ice insert gw_logs.test4 -p file:///home/aadant/test.parquet
2025-06-25 11:13:08 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: processing
2025-06-25 11:13:08 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: jvm: heap: init=3008M,used=16M,committed=224M,free=143599M,max=143616M, gc pressure: 100.00%(14ms), off-heap: init=7M,used=51M,committed=53M, open fds: 11
2025-06-25 11:13:08 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > file:///home/aadant/test.parquet: copying to s3://warehouse/test/test/data/1750867988512-06a056686f5853ecfc8ab14ec56fc10fee707ecbe9b929a3b53236991e9fee4f.parquet
2025-06-25 11:13:08 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.io.IOException: Error processing file(s)
	at com.altinity.ice.cli.internal.cmd.Insert.run(Insert.java:264)
	at com.altinity.ice.cli.Main.insert(Main.java:317)
	at com.altinity.ice.cli.Main.lambda$main$1(Main.java:480) [9 skipped]
	at com.altinity.ice.cli.Main.main(Main.java:487) [1 skipped]
Caused by: java.io.IOException: Error processing file:///home/aadant/test.parquet
	at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$3(Insert.java:249)
 [4 skipped]
Caused by: java.lang.UnsupportedOperationException: Unsupported unit for timestamp: NANOS
	at org.apache.iceberg.data.parquet.BaseParquetReaders.timestampReader(BaseParquetReaders.java:128)
	at org.apache.iceberg.data.parquet.BaseParquetReaders$LogicalTypeReadBuilder.visit(BaseParquetReaders.java:211)
	at org.apache.parquet.schema.LogicalTypeAnnotation$TimestampLogicalTypeAnnotation.accept(LogicalTypeAnnotation.java:710)
	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.primitive(BaseParquetReaders.java:394)
	at org.apache.iceberg.data.parquet.BaseParquetReaders$ReadBuilder.primitive(BaseParquetReaders.java:253)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visit(TypeWithSchemaVisitor.java:52)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visitField(TypeWithSchemaVisitor.java:183)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visitFields(TypeWithSchemaVisitor.java:198)
	at org.apache.iceberg.parquet.TypeWithSchemaVisitor.visit(TypeWithSchemaVisitor.java:47)
	at org.apache.iceberg.data.parquet.BaseParquetReaders.createReader(BaseParquetReaders.java:69)
	at org.apache.iceberg.data.parquet.BaseParquetReaders.createReader(BaseParquetReaders.java:61)
	at org.apache.iceberg.data.parquet.GenericParquetReaders.buildReader(GenericParquetReaders.java:51)
	at com.altinity.ice.cli.internal.cmd.Insert.lambda$processFile$5(Insert.java:468)
	at org.apache.iceberg.parquet.ReadConf.<init>(ReadConf.java:121)
	at org.apache.iceberg.parquet.ParquetReader.init(ParquetReader.java:71)
	at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:91)
	at org.apache.iceberg.parquet.ParquetReader.iterator(ParquetReader.java:37)
	at org.apache.iceberg.io.FileAppender.addAll(FileAppender.java:36)
	at com.altinity.ice.cli.internal.cmd.Insert.copy(Insert.java:713)
	at com.altinity.ice.cli.internal.cmd.Insert.processFile(Insert.java:478)
	at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$3(Insert.java:228)
	... 4 common frames omitted

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions