[SPARK-53420][BUILD] Upgrade Parquet to 1.16.0 by pan3793 · Pull Request #52165 · apache/spark

pan3793 · 2025-08-28T16:08:08Z

What changes were proposed in this pull request?

Parquet Java 1.16.0 Release Notes: https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.16.0

Why are the changes needed?

Keep Parquet update to date, benefit from upstream bugfixes and improvements.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GHA.

Run TPC-DS 300G (query time in seconds), no surprise compared to Parquet 1.15.2

query_name	parquet 1.15.2	parquet 1.16.0 RC2
q01.sql	31	29
q02.sql	26	20
q03.sql	9	9
q04.sql	67	67
q05.sql	14	14
q06.sql	8	8
q07.sql	5	5
q08.sql	6	6
q09.sql	13	12
q10.sql	6	6
q11.sql	31	31
q12.sql	4	3
q13.sql	5	5
q14a.sql	52	53
q14b.sql	41	41
q15.sql	4	4
q16.sql	16	16
q17.sql	7	7
q18.sql	6	6
q19.sql	6	6
q20.sql	3	3
q21.sql	5	4
q22.sql	10	10
q23a.sql	76	74
q23b.sql	80	80
q24a.sql	39	39
q24b.sql	32	31
q25.sql	7	7
q26.sql	4	3
q27.sql	3	3
q28.sql	17	16
q29.sql	9	9
q30.sql	9	9
q31.sql	13	13
q32.sql	2	2
q33.sql	5	6
q34.sql	5	5
q35.sql	13	13
q36.sql	4	4
q37.sql	7	6
q38.sql	13	12
q39a.sql	7	7
q39b.sql	7	6
q40.sql	5	6
q41.sql	0	0
q42.sql	1	2
q43.sql	4	4
q44.sql	7	7
q45.sql	3	3
q46.sql	7	7
q47.sql	11	12
q48.sql	5	5
q49.sql	7	7
q50.sql	16	16
q51.sql	15	14
q52.sql	2	2
q53.sql	2	2
q54.sql	4	5
q55.sql	1	2
q56.sql	4	4
q57.sql	12	9
q58.sql	4	6
q59.sql	14	10
q60.sql	7	7
q61.sql	4	3
q62.sql	4	5
q63.sql	2	2
q64.sql	21	21
q65.sql	12	12
q66.sql	6	7
q67.sql	46	45
q68.sql	7	6
q69.sql	10	10
q70.sql	5	5
q71.sql	6	6
q72.sql	17	16
q73.sql	4	4
q74.sql	24	24
q75.sql	24	21
q76.sql	12	10
q77.sql	6	5
q78.sql	31	31
q79.sql	5	5
q80.sql	9	8
q81.sql	8	7
q82.sql	11	11
q83.sql	4	4
q84.sql	5	4
q85.sql	8	7
q86.sql	3	3
q87.sql	14	13
q88.sql	13	13
q89.sql	4	3
q90.sql	4	3
q91.sql	3	4
q92.sql	2	2
q93.sql	21	20
q94.sql	9	9
q95.sql	62	61
q96.sql	5	3
q97.sql	12	11
q98.sql	4	3
q99.sql	6	6

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun

Thank you always, @pan3793 .

Fokko · 2025-08-28T17:15:35Z

You beat me to it @pan3793 Thanks for creating this PR 🚀

pan3793 · 2025-08-28T17:21:23Z

@Fokko, I'm looking forward to this release for a long time, since it includes my two patches required by SPARK-52011(#50765) 😄

pan3793 · 2025-08-28T17:34:36Z

@dongjoon-hyun @Fokko, this release contains a correctness fix, it affects the Parquet 1.15.2 used by Spark branch-4.0.

Data issue after the optimization of bloom_filter_length parquet-java#3249

It seems that the Parquet community does not have a plan for another release of 1.15.x yet, not sure if this will be a blocker for the upcoming Spark 4.0.1

dongjoon-hyun · 2025-08-28T17:38:00Z

Thank you for informing me, @pan3793 .

pan3793 · 2025-08-28T19:02:59Z

One test failed due to the Parquet data file size change, it's not a real issue, I opened #52168 to improve the test.

dongjoon-hyun · 2025-08-28T19:15:45Z

I took a look at #52168, but we had better update the expectSize instead of removing a test coverage, @pan3793 .

dongjoon-hyun · 2025-09-01T04:36:13Z

Thank you so much always for keeping tracking the upstream RCs, @pan3793 .

As a release manager of Apache Spark 4.0.1, as of now, I don't think this is a blocker for Apache Spark 4.0.1 release because

This is a feature release instead of a maintenance release of Parquet (as you mentioned).
There is no official Parquet release (still RC3?).
After we get the official 1.16.0 release, we still need to spend more efforts to validate more with Apache Spark 4.1.0 (master branch) for a while because Apache Parquet is the default file format of Apache Spark 4.1.0 first.

Of course, after we merge this to master branch, all downstream forks can cherry-pick and use Apache Parquet 1.16.0 at their own risks in their production environment. That's the main purpose to valid and merge this PR to master branch. Personally, I'm also going to verify with internal benchmarks and more production usages if Apache Parquet 1.16.0 RC3 passes the vote.

I'm sure that we agree that the best case for all and world-wide community, Apache Parquet 1.15.3 is released for Apache Spark 4.0.2 in next 3 months.

pan3793 · 2025-09-01T04:50:56Z

@dongjoon-hyun, I agree with your decision, please continue the 4.0.1 release with the existing Parquet 1.15.2

dongjoon-hyun · 2025-09-01T22:00:28Z

Thank you so much again.

pan3793 · 2025-09-02T09:28:06Z

Have run TPC-DS 300G internally, no regression found, I will vote +1 for Parquet 1.16.0 RC2.

dongjoon-hyun · 2025-09-02T16:33:39Z

Have run TPC-DS 300G internally, no regression found, I will vote +1 for Parquet 1.16.0 RC2.

Thank you for adding the result. Could you elaborate about the environment a little more? Is it against master branch with Apache Spark 4.1.0-SNAPSHOT and Hadoop 3.4.2 and S3 data (or local files) on a single disk, @pan3793 ?

pan3793 · 2025-09-02T16:42:12Z

Is it against master branch with Apache Spark 4.1.0-SNAPSHOT

Exactly, I built OSS Spark without any changes. (so Spark uses Hadoop 3.4.2 client)

... and Hadoop 3.4.2 and S3 data (or local files) on a single disk

Runs in a small YARN cluster, data is stored in HDFS. YARN/HDFS server version is 3.3.6, both shuffle and HDFS use HDD.

dongjoon-hyun · 2025-09-02T16:53:19Z

Thank you so much for the details.

pan3793 · 2025-09-03T10:08:41Z

@dongjoon-hyun this should be ready to go. also cc @wangyum

dongjoon-hyun

+1, LGTM for Apache Spark 4.1.0.

Thank you, @pan3793 and @wangyum .

Let's merge this to master branch in order to get more chances of community testings.

LuciferYang · 2025-09-04T03:55:37Z

late LGTM

### What changes were proposed in this pull request? Parquet Java 1.16.0 Release Notes: https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.16.0 ### Why are the changes needed? Keep Parquet update to date, benefit from upstream bugfixes and improvements. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GHA. Run TPC-DS 300G (query time in seconds), no surprise compared to Parquet 1.15.2 | query_name | parquet 1.15.2 | parquet 1.16.0 RC2 | | :--------- | :----------- | :----------- | | q01.sql | 31 | 29 | | q02.sql | 26 | 20 | | q03.sql | 9 | 9 | | q04.sql | 67 | 67 | | q05.sql | 14 | 14 | | q06.sql | 8 | 8 | | q07.sql | 5 | 5 | | q08.sql | 6 | 6 | | q09.sql | 13 | 12 | | q10.sql | 6 | 6 | | q11.sql | 31 | 31 | | q12.sql | 4 | 3 | | q13.sql | 5 | 5 | | q14a.sql | 52 | 53 | | q14b.sql | 41 | 41 | | q15.sql | 4 | 4 | | q16.sql | 16 | 16 | | q17.sql | 7 | 7 | | q18.sql | 6 | 6 | | q19.sql | 6 | 6 | | q20.sql | 3 | 3 | | q21.sql | 5 | 4 | | q22.sql | 10 | 10 | | q23a.sql | 76 | 74 | | q23b.sql | 80 | 80 | | q24a.sql | 39 | 39 | | q24b.sql | 32 | 31 | | q25.sql | 7 | 7 | | q26.sql | 4 | 3 | | q27.sql | 3 | 3 | | q28.sql | 17 | 16 | | q29.sql | 9 | 9 | | q30.sql | 9 | 9 | | q31.sql | 13 | 13 | | q32.sql | 2 | 2 | | q33.sql | 5 | 6 | | q34.sql | 5 | 5 | | q35.sql | 13 | 13 | | q36.sql | 4 | 4 | | q37.sql | 7 | 6 | | q38.sql | 13 | 12 | | q39a.sql | 7 | 7 | | q39b.sql | 7 | 6 | | q40.sql | 5 | 6 | | q41.sql | 0 | 0 | | q42.sql | 1 | 2 | | q43.sql | 4 | 4 | | q44.sql | 7 | 7 | | q45.sql | 3 | 3 | | q46.sql | 7 | 7 | | q47.sql | 11 | 12 | | q48.sql | 5 | 5 | | q49.sql | 7 | 7 | | q50.sql | 16 | 16 | | q51.sql | 15 | 14 | | q52.sql | 2 | 2 | | q53.sql | 2 | 2 | | q54.sql | 4 | 5 | | q55.sql | 1 | 2 | | q56.sql | 4 | 4 | | q57.sql | 12 | 9 | | q58.sql | 4 | 6 | | q59.sql | 14 | 10 | | q60.sql | 7 | 7 | | q61.sql | 4 | 3 | | q62.sql | 4 | 5 | | q63.sql | 2 | 2 | | q64.sql | 21 | 21 | | q65.sql | 12 | 12 | | q66.sql | 6 | 7 | | q67.sql | 46 | 45 | | q68.sql | 7 | 6 | | q69.sql | 10 | 10 | | q70.sql | 5 | 5 | | q71.sql | 6 | 6 | | q72.sql | 17 | 16 | | q73.sql | 4 | 4 | | q74.sql | 24 | 24 | | q75.sql | 24 | 21 | | q76.sql | 12 | 10 | | q77.sql | 6 | 5 | | q78.sql | 31 | 31 | | q79.sql | 5 | 5 | | q80.sql | 9 | 8 | | q81.sql | 8 | 7 | | q82.sql | 11 | 11 | | q83.sql | 4 | 4 | | q84.sql | 5 | 4 | | q85.sql | 8 | 7 | | q86.sql | 3 | 3 | | q87.sql | 14 | 13 | | q88.sql | 13 | 13 | | q89.sql | 4 | 3 | | q90.sql | 4 | 3 | | q91.sql | 3 | 4 | | q92.sql | 2 | 2 | | q93.sql | 21 | 20 | | q94.sql | 9 | 9 | | q95.sql | 62 | 61 | | q96.sql | 5 | 3 | | q97.sql | 12 | 11 | | q98.sql | 4 | 3 | | q99.sql | 6 | 6 | ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52165 from pan3793/parquet-1.16.0. Authored-by: Cheng Pan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…odError issue see apache/spark#52165

## Changes | Cause | Type | Category | Description | Affected Files | |-------|------|----------|-------------|----------------| | N/A | Feat | Build | Update build configuration to support Spark 4.1 UT | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/pom.xml`, `gluten-ut/spark41/pom.xml`, `tools/gluten-it/pom.xml` | | [#52165](apache/spark#52165) | Fix | Dependency | Update Parquet dependency version to 1.16.0 to avoid NoSuchMethodError issue | `gluten-ut/spark41/pom.xml` | | [#51477](apache/spark#51477) | Fix | Compatibility | Update imports to reflect streaming runtime package refactoring in Apache Spark | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala`, `gluten-ut/spark41/.../GlutenStreamingQuerySuite.scala` | | [#50674](apache/spark#50674) | Fix | Compatibility | Fix compatibility issue introduced by `TypedConfigBuilder` | `gluten-substrait/.../ExpressionConverter.scala`, `gluten-ut/spark41/.../GlutenCSVSuite.scala`, `gluten-ut/spark41/.../GlutenJsonSuite.scala` | | [#49766](apache/spark#49766) | Fix | Compatibility | Disable V2 bucketing in GlutenDynamicPartitionPruningSuite since spark.sql.sources.v2.bucketing.enabled is now enabled by default | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala` | | [#42414](apache/spark#42414), [#53038](apache/spark#53038) | Fix | Bug Fix | Resolve an issue introduced by SPARK-42414, as identified in SPARK-53038 | `backends-velox/.../VeloxBloomFilterAggregate.scala` | | N/A | Fix | Bug Fix | Enforce row fallback for unsupported cached batches - keep columnar execution only when schema validation succeeds | `backends-velox/.../ColumnarCachedBatchSerializer.scala` | | [SPARK-53132](apache/spark#53132), [SPARK-53142](apache/spark#53142) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 KeyGroupedPartitioningSuite tests. Excluded tests: `SPARK-53322*`, `SPARK-54439*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [SPARK-53535](https://issues.apache.org/jira/browse/SPARK-53535), [SPARK-54220](https://issues.apache.org/jira/browse/SPARK-54220) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenParquetIOSuite tests. Excluded tests: `SPARK-53535*`, `vectorized reader: missing all struct fields*`, `SPARK-54220*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52645](apache/spark#52645) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenStreamingQuerySuite tests. Excluded tests: `SPARK-53942: changing the number of stateless shuffle partitions via config`, `SPARK-53942: stateful shuffle partitions are retained from old checkpoint` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#47856](apache/spark#47856) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuite and GlutenJoinSuite tests. Excluded tests: `SPARK-49386: Window spill with more than the inMemoryThreshold and spillSizeThreshold`, `SPARK-49386: test SortMergeJoin (with spill by size threshold)` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52157](apache/spark#52157) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests. Excluded test: `#53413: Cleanup shuffle dependencies for commands` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#48470](apache/spark#48470) | 4.1.0 | Test Exclusion | Exclude split test in GlutenRegexpExpressionsSuite. Excluded test: `GlutenRegexpExpressionsSuite.SPLIT` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#51623](apache/spark#51623) | 4.1.0 | Test Exclusion | Add `spark.sql.unionOutputPartitioning=false` to Maven test args. Excluded tests: `GlutenBroadcastExchangeSuite.SPARK-52962`, `GlutenDataFrameSetOperationsSuite.SPARK-52921*` | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/spark41/.../VeloxTestSettings.scala`, `tools/gluten-it/common/.../Suite.scala` | | N/A | 4.1.0 | Test Exclusion | Excludes failed SQL tests that need to be fixed for Spark 4.1 compatibility. Excluded tests: `decimalArithmeticOperations.sql`, `identifier-clause.sql`, `keywords.sql`, `literals.sql`, `operators.sql`, `exists-orderby-limit.sql`, `postgreSQL/date.sql`, `nonansi/keywords.sql`, `nonansi/literals.sql`, `datetime-legacy.sql`, `datetime-parsing-invalid.sql`, `misc-functions.sql` | `gluten-ut/spark41/.../VeloxSQLQueryTestSettings.scala` |

## Changes | Cause | Type | Category | Description | Affected Files | |-------|------|----------|-------------|----------------| | N/A | Feat | Build | Update build configuration to support Spark 4.1 UT | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/pom.xml`, `gluten-ut/spark41/pom.xml`, `tools/gluten-it/pom.xml` | | [#52165](apache/spark#52165) | Fix | Dependency | Update Parquet dependency version to 1.16.0 to avoid NoSuchMethodError issue | `gluten-ut/spark41/pom.xml` | | [#51477](apache/spark#51477) | Fix | Compatibility | Update imports to reflect streaming runtime package refactoring in Apache Spark | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala`, `gluten-ut/spark41/.../GlutenStreamingQuerySuite.scala` | | [#50674](apache/spark#50674) | Fix | Compatibility | Fix compatibility issue introduced by `TypedConfigBuilder` | `gluten-substrait/.../ExpressionConverter.scala`, `gluten-ut/spark41/.../GlutenCSVSuite.scala`, `gluten-ut/spark41/.../GlutenJsonSuite.scala` | | [#49766](apache/spark#49766) | Fix | Compatibility | Disable V2 bucketing in GlutenDynamicPartitionPruningSuite since spark.sql.sources.v2.bucketing.enabled is now enabled by default | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala` | | [#42414](apache/spark#42414), [#53038](apache/spark#53038) | Fix | Bug Fix | Resolve an issue introduced by SPARK-42414, as identified in SPARK-53038 | `backends-velox/.../VeloxBloomFilterAggregate.scala` | | N/A | Fix | Bug Fix | Enforce row fallback for unsupported cached batches - keep columnar execution only when schema validation succeeds | `backends-velox/.../ColumnarCachedBatchSerializer.scala` | | [SPARK-53132](apache/spark#53132), [SPARK-53142](apache/spark#53142) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 KeyGroupedPartitioningSuite tests. Excluded tests: `SPARK-53322*`, `SPARK-54439*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [SPARK-53535](https://issues.apache.org/jira/browse/SPARK-53535), [SPARK-54220](https://issues.apache.org/jira/browse/SPARK-54220) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenParquetIOSuite tests. Excluded tests: `SPARK-53535*`, `vectorized reader: missing all struct fields*`, `SPARK-54220*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52645](apache/spark#52645) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenStreamingQuerySuite tests. Excluded tests: `SPARK-53942: changing the number of stateless shuffle partitions via config`, `SPARK-53942: stateful shuffle partitions are retained from old checkpoint` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#47856](apache/spark#47856) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuite and GlutenJoinSuite tests. Excluded tests: `SPARK-49386: Window spill with more than the inMemoryThreshold and spillSizeThreshold`, `SPARK-49386: test SortMergeJoin (with spill by size threshold)` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52157](apache/spark#52157) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests. Excluded test: `#53413: Cleanup shuffle dependencies for commands` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#48470](apache/spark#48470) | 4.1.0 | Test Exclusion | Exclude split test in GlutenRegexpExpressionsSuite. Excluded test: `GlutenRegexpExpressionsSuite.SPLIT` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#51623](apache/spark#51623) | 4.1.0 | Test Exclusion | Add `spark.sql.unionOutputPartitioning=false` to Maven test args. Excluded tests: `GlutenBroadcastExchangeSuite.SPARK-52962`, `GlutenDataFrameSetOperationsSuite.SPARK-52921*` | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/spark41/.../VeloxTestSettings.scala`, `tools/gluten-it/common/.../Suite.scala` | | N/A | 4.1.0 | Test Exclusion | Excludes failed SQL tests that need to be fixed for Spark 4.1 compatibility. Excluded tests: `decimalArithmeticOperations.sql`, `identifier-clause.sql`, `keywords.sql`, `literals.sql`, `operators.sql`, `exists-orderby-limit.sql`, `postgreSQL/date.sql`, `nonansi/keywords.sql`, `nonansi/literals.sql`, `datetime-legacy.sql`, `datetime-parsing-invalid.sql`, `misc-functions.sql` | `gluten-ut/spark41/.../VeloxSQLQueryTestSettings.scala` | | apache#11252 | 4.1.0 | Test Exclusion | Exclude Gluten test for SPARK-47939: Explain should work with parameterized queries | `gluten-ut/spark41/.../VeloxTestSettings.scala` |

## Changes | Cause | Type | Category | Description | Affected Files | |-------|------|----------|-------------|----------------| | N/A | Feat | Build | Update build configuration to support Spark 4.1 UT | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/pom.xml`, `gluten-ut/spark41/pom.xml`, `tools/gluten-it/pom.xml` | | [#52165](apache/spark#52165) | Fix | Dependency | Update Parquet dependency version to 1.16.0 to avoid NoSuchMethodError issue | `gluten-ut/spark41/pom.xml` | | [#51477](apache/spark#51477) | Fix | Compatibility | Update imports to reflect streaming runtime package refactoring in Apache Spark | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala`, `gluten-ut/spark41/.../GlutenStreamingQuerySuite.scala` | | [#50674](apache/spark#50674) | Fix | Compatibility | Fix compatibility issue introduced by `TypedConfigBuilder` | `gluten-substrait/.../ExpressionConverter.scala`, `gluten-ut/spark41/.../GlutenCSVSuite.scala`, `gluten-ut/spark41/.../GlutenJsonSuite.scala` | | [#49766](apache/spark#49766) | Fix | Compatibility | Disable V2 bucketing in GlutenDynamicPartitionPruningSuite since spark.sql.sources.v2.bucketing.enabled is now enabled by default | `gluten-ut/spark41/.../GlutenDynamicPartitionPruningSuite.scala` | | [#42414](apache/spark#42414), [#53038](apache/spark#53038) | Fix | Bug Fix | Resolve an issue introduced by SPARK-42414, as identified in SPARK-53038 | `backends-velox/.../VeloxBloomFilterAggregate.scala` | | N/A | Fix | Bug Fix | Enforce row fallback for unsupported cached batches - keep columnar execution only when schema validation succeeds | `backends-velox/.../ColumnarCachedBatchSerializer.scala` | | [SPARK-53132](apache/spark#53132), [SPARK-53142](apache/spark#53142) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 KeyGroupedPartitioningSuite tests. Excluded tests: `SPARK-53322*`, `SPARK-54439*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [SPARK-53535](https://issues.apache.org/jira/browse/SPARK-53535), [SPARK-54220](https://issues.apache.org/jira/browse/SPARK-54220) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenParquetIOSuite tests. Excluded tests: `SPARK-53535*`, `vectorized reader: missing all struct fields*`, `SPARK-54220*` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52645](apache/spark#52645) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenStreamingQuerySuite tests. Excluded tests: `SPARK-53942: changing the number of stateless shuffle partitions via config`, `SPARK-53942: stateful shuffle partitions are retained from old checkpoint` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#47856](apache/spark#47856) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenDataFrameWindowFunctionsSuite and GlutenJoinSuite tests. Excluded tests: `SPARK-49386: Window spill with more than the inMemoryThreshold and spillSizeThreshold`, `SPARK-49386: test SortMergeJoin (with spill by size threshold)` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#52157](apache/spark#52157) | 4.1.0 | Test Exclusion | Exclude additional Spark 4.1 GlutenQueryExecutionSuite tests. Excluded test: `#53413: Cleanup shuffle dependencies for commands` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#48470](apache/spark#48470) | 4.1.0 | Test Exclusion | Exclude split test in GlutenRegexpExpressionsSuite. Excluded test: `GlutenRegexpExpressionsSuite.SPLIT` | `gluten-ut/spark41/.../VeloxTestSettings.scala` | | [#51623](apache/spark#51623) | 4.1.0 | Test Exclusion | Add `spark.sql.unionOutputPartitioning=false` to Maven test args. Excluded tests: `GlutenBroadcastExchangeSuite.SPARK-52962`, `GlutenDataFrameSetOperationsSuite.SPARK-52921*` | `.github/workflows/velox_backend_x86.yml`, `gluten-ut/spark41/.../VeloxTestSettings.scala`, `tools/gluten-it/common/.../Suite.scala` | | N/A | 4.1.0 | Test Exclusion | Excludes failed SQL tests that need to be fixed for Spark 4.1 compatibility. Excluded tests: `decimalArithmeticOperations.sql`, `identifier-clause.sql`, `keywords.sql`, `literals.sql`, `operators.sql`, `exists-orderby-limit.sql`, `postgreSQL/date.sql`, `nonansi/keywords.sql`, `nonansi/literals.sql`, `datetime-legacy.sql`, `datetime-parsing-invalid.sql`, `misc-functions.sql` | `gluten-ut/spark41/.../VeloxSQLQueryTestSettings.scala` | | #11252 | 4.1.0 | Test Exclusion | Exclude Gluten test for SPARK-47939: Explain should work with parameterized queries | `gluten-ut/spark41/.../VeloxTestSettings.scala` |

github-actions bot added the BUILD label Aug 28, 2025

dongjoon-hyun reviewed Aug 28, 2025

View reviewed changes

[SPARK-53420][BUILD] Upgrade Parquet to 1.16.0 RC0

1fb7939

pan3793 force-pushed the parquet-1.16.0 branch from 3f8bb45 to 1fb7939 Compare September 1, 2025 03:48

remove staging repo

b54dbcc

pan3793 marked this pull request as ready for review September 3, 2025 03:06

wangyum approved these changes Sep 4, 2025

View reviewed changes

dongjoon-hyun approved these changes Sep 4, 2025

View reviewed changes

dongjoon-hyun closed this in 65c5775 Sep 4, 2025

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[Fix] Update Parquet dependency version to 1.16.0 to avoid NoSuchMeth…

c381112

…odError issue see apache/spark#52165

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[Fix] Update Parquet dependency version to 1.16.0 to avoid NoSuchMeth…

a66f12f

…odError issue see apache/spark#52165

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 5, 2026

[Fix] Update Parquet dependency version to 1.16.0 to avoid NoSuchMeth…

1c105cf

…odError issue see apache/spark#52165

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 5, 2026

[Fix] Update Parquet dependency version to 1.16.0 to avoid NoSuchMeth…

3d2cf0e

…odError issue see apache/spark#52165

baibaichen mentioned this pull request Jan 7, 2026

[GLUTEN-11343][CORE][VL] Support Spark 4.1 UT apache/gluten#11353

Merged

Conversation

pan3793 commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Fokko commented Aug 28, 2025

Uh oh!

pan3793 commented Aug 28, 2025

Uh oh!

pan3793 commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Aug 28, 2025

Uh oh!

pan3793 commented Aug 28, 2025

Uh oh!

dongjoon-hyun commented Aug 28, 2025

Uh oh!

dongjoon-hyun commented Sep 1, 2025

Uh oh!

pan3793 commented Sep 1, 2025

Uh oh!

dongjoon-hyun commented Sep 1, 2025

Uh oh!

pan3793 commented Sep 2, 2025

Uh oh!

dongjoon-hyun commented Sep 2, 2025

Uh oh!

pan3793 commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Sep 2, 2025

Uh oh!

pan3793 commented Sep 3, 2025

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Sep 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pan3793 commented Aug 28, 2025 •

edited

Loading

pan3793 commented Aug 28, 2025 •

edited

Loading

pan3793 commented Sep 2, 2025 •

edited

Loading