[SPARK-51812][SQL] Remove redundant parameters of some methods in `QueryExecution` by panbingkun · Pull Request #50598 · apache/spark

panbingkun · 2025-04-16T02:11:06Z

What changes were proposed in this pull request?

The pr aims to remove redundant parameters of some methods in QueryExecution, includes:

def createSparkPlan(~~sparkSession: SparkSession~~, planner: SparkPlanner, plan: LogicalPlan): SparkPlan
def prepareExecutedPlan(~~sparkSession: SparkSession~~, plan: LogicalPlan, context: AdaptiveExecutionContext): SparkPlan

Why are the changes needed?

The parameters sparkSession of the method QueryExecution#createSparkPlan are not actually used in the method body
The parameters sparkSession of the method QueryExecution#prepareExecutedPlan is actually the attribute session of parameter context
These have brought misunderstanding and burden on the developers.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No.

…ion#createSparkPlan`

panbingkun · 2025-04-16T02:13:42Z

cc @cloud-fan @LuciferYang

LuciferYang

+1, LGTM
cc @cloud-fan

summaryzb · 2025-04-16T02:49:37Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

@@ -606,7 +605,7 @@ object QueryExecution {
      session: SparkSession,


This session can be removed too, since it's the same SparkSession of the AdaptiveExecutionContext

Yeah, that's right.

LuciferYang · 2025-04-17T02:25:03Z

Merged into master. Thanks @panbingkun @HyukjinKwon @summaryzb

see apache/spark#50598

| Cause | Type | Category | Description | Affected Files | |-------|------|----------|-------------|----------------| | - | Feat | Feature | Introduce Spark41Shims and update build configuration to support Spark 4.1. | pom.xml shims/pom.xml shims/spark41/pom.xml shims/spark41/.../META-INF/services/org.apache.gluten.sql.shims.SparkShimProvider shims/spark41/.../spark41/Spark41Shims.scala shims/spark41/.../spark41/SparkShimProvider.scala | | [#51477](apache/spark#51477) | Fix | Compatibility | Use class name instead of class object for streaming call detection to ensure Spark 4.1 compatibility. | gluten-core/.../caller/CallerInfo.scala | | [#50852](apache/spark#50852) | Fix | Compatibility | Add printOutputColumns parameter to generateTreeString methods | shims/spark41/.../GenerateTreeStringShim.scala | | [#51775](apache/spark#51775) | Fix | Compatibility | Remove unused MDC import in FileSourceScanExecShim.scala | shims/spark41/.../FileSourceScanExecShim.scala | | [#51979](apache/spark#51979) | Fix | Compatibility | Add missing StoragePartitionJoinParams import in BatchScanExecShim and AbstractBatchScanExec | shims/spark41/.../v2/AbstractBatchScanExec.scala shims/spark41/.../v2/BatchScanExecShim.scala | | [#51302](apache/spark#51302) | Fix | Compatibility | Remove TimeAdd from ExpressionConverter and ExpressionMappings for test | gluten-substrait/.../ExpressionConverter.scala gluten-substrait/.../ExpressionMappings.scala | | [#50598](apache/spark#50598) | Fix | Compatibility | Adapt to QueryExecution.createSparkPlan interface change | gluten-substrait/.../GlutenImplicits.scala shims/spark*/.../QueryExecutionShim.scala | | [#52599](apache/spark#52599) | Fix | Compatibility | Adapt to DataSourceV2Relation interface change | backends-velox/.../ArrowConvertorRule.scala shims/spark*/.../v2/DataSourceV2RelationShim.scala | | [#52384](apache/spark#52384) | Fix | Compatibility | Using new interface of ParquetFooterReader | backends-velox/.../ParquetMetadataUtils.scala gluten-ut/spark40/.../parquet/GlutenParquetRowIndexSuite.scala shims/spark*/.../parquet/ParquetFooterReaderShim.scala | | [#52509](apache/spark#52509) | Fix | Build | Update Scala version to 2.13.17 in pom.xml to fix `java.lang.NoSuchMethodError: 'java.lang.String scala.util.hashing.MurmurHash3$.caseClassHash$default$2()'` | pom.xml | | - | Fix | Test | Refactor Spark version checks in VeloxHashJoinSuite to improve readability and maintainability | backends-velox/.../VeloxHashJoinSuite.scala | | [#50849](apache/spark#50849) | Fix | Test | Fix MiscOperatorSuite to support OneRowRelationExec plan Spark 4.1 | backends-velox/.../MiscOperatorSuite.scala | | [#52723](apache/spark#52723) | Fix | Compatibility | Add GeographyVal and GeometryVal support in ColumnarArrayShim | shims/spark41/.../vectorized/ColumnarArrayShim.java | | [#48470](apache/spark#48470) | 4.1.0 | Exclude | Exclude split test in VeloxStringFunctionsSuite | backends-velox/.../VeloxStringFunctionsSuite.scala | | [#51259](apache/spark#51259) | 4.1.0 | Exclude | Only Run ArrowEvalPythonExecSuite tests up to Spark 4.0， we need update ci python to 3.10 | backends-velox/.../python/ArrowEvalPythonExecSuite.scala |

| Cause | Type | Category | Description | Affected Files | |-------|------|----------|-------------|----------------| | - | Feat | Feature | Introduce Spark41Shims and update build configuration to support Spark 4.1. | pom.xml shims/pom.xml shims/spark41/pom.xml shims/spark41/.../META-INF/services/org.apache.gluten.sql.shims.SparkShimProvider shims/spark41/.../spark41/Spark41Shims.scala shims/spark41/.../spark41/SparkShimProvider.scala | | [#51477](apache/spark#51477) | Fix | Compatibility | Use class name instead of class object for streaming call detection to ensure Spark 4.1 compatibility. | gluten-core/.../caller/CallerInfo.scala | | [#50852](apache/spark#50852) | Fix | Compatibility | Add printOutputColumns parameter to generateTreeString methods | shims/spark41/.../GenerateTreeStringShim.scala | | [#51775](apache/spark#51775) | Fix | Compatibility | Remove unused MDC import in FileSourceScanExecShim.scala | shims/spark41/.../FileSourceScanExecShim.scala | | [#51979](apache/spark#51979) | Fix | Compatibility | Add missing StoragePartitionJoinParams import in BatchScanExecShim and AbstractBatchScanExec | shims/spark41/.../v2/AbstractBatchScanExec.scala shims/spark41/.../v2/BatchScanExecShim.scala | | [#51302](apache/spark#51302) | Fix | Compatibility | Remove TimeAdd from ExpressionConverter and ExpressionMappings for test | gluten-substrait/.../ExpressionConverter.scala gluten-substrait/.../ExpressionMappings.scala | | [#50598](apache/spark#50598) | Fix | Compatibility | Adapt to QueryExecution.createSparkPlan interface change | gluten-substrait/.../GlutenImplicits.scala shims/spark\*/.../shims/spark\*/Spark*Shims.scala | | [#52599](apache/spark#52599) | Fix | Compatibility | Adapt to DataSourceV2Relation interface change | backends-velox/.../ArrowConvertorRule.scala | | [#52384](apache/spark#52384) | Fix | Compatibility | Using new interface of ParquetFooterReader | backends-velox/.../ParquetMetadataUtils.scala gluten-ut/spark40/.../parquet/GlutenParquetRowIndexSuite.scala shims/spark*/.../parquet/ParquetFooterReaderShim.scala | | [#52509](apache/spark#52509) | Fix | Build | Update Scala version to 2.13.17 in pom.xml to fix `java.lang.NoSuchMethodError: 'java.lang.String scala.util.hashing.MurmurHash3$.caseClassHash$default$2()'` | pom.xml | | - | Fix | Test | Refactor Spark version checks in VeloxHashJoinSuite to improve readability and maintainability | backends-velox/.../VeloxHashJoinSuite.scala | | [#50849](apache/spark#50849) | Fix | Test | Fix MiscOperatorSuite to support OneRowRelationExec plan Spark 4.1 | backends-velox/.../MiscOperatorSuite.scala | | [#52723](apache/spark#52723) | Fix | Compatibility | Add GeographyVal and GeometryVal support in ColumnarArrayShim | shims/spark41/.../vectorized/ColumnarArrayShim.java | | [#48470](apache/spark#48470) | 4.1.0 | Exclude | Exclude split test in VeloxStringFunctionsSuite | backends-velox/.../VeloxStringFunctionsSuite.scala | | [#51259](apache/spark#51259) | 4.1.0 | Exclude | Only Run ArrowEvalPythonExecSuite tests up to Spark 4.0， we need update ci python to 3.10 | backends-velox/.../python/ArrowEvalPythonExecSuite.scala |

[SPARK-51812][SQL] Remove redundant parameters of method `QueryExecut…

6705fc5

…ion#createSparkPlan`

github-actions bot added the SQL label Apr 16, 2025

LuciferYang approved these changes Apr 16, 2025

View reviewed changes

HyukjinKwon approved these changes Apr 16, 2025

View reviewed changes

summaryzb reviewed Apr 16, 2025

View reviewed changes

update

c0b9f18

panbingkun changed the title ~~[SPARK-51812][SQL] Remove redundant parameters of method QueryExecution#createSparkPlan~~ [SPARK-51812][SQL] Remove redundant parameters of some methods in QueryExecution Apr 16, 2025

summaryzb approved these changes Apr 16, 2025

View reviewed changes

LuciferYang closed this in ed702c0 Apr 16, 2025

baibaichen added a commit to apache/gluten that referenced this pull request Dec 31, 2025

[Fix] Adapt to QueryExecution.createSparkPlan interface change

939dac2

see apache/spark#50598

baibaichen added a commit to baibaichen/gluten that referenced this pull request Dec 31, 2025

[Fix] Adapt to QueryExecution.createSparkPlan interface change

4576a50

see apache/spark#50598

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[Fix] Adapt to QueryExecution.createSparkPlan interface change

ab647ce

see apache/spark#50598

baibaichen added a commit to baibaichen/gluten that referenced this pull request Jan 4, 2026

[Fix] Adapt to QueryExecution.createSparkPlan interface change

1855fe4

see apache/spark#50598

baibaichen mentioned this pull request Jan 5, 2026

[GLUTEN-11346][CORE][VL] Add Spark 4.1 Shim Layer apache/gluten#11347

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-51812][SQL] Remove redundant parameters of some methods in `QueryExecution`#50598

[SPARK-51812][SQL] Remove redundant parameters of some methods in `QueryExecution`#50598
panbingkun wants to merge 2 commits intoapache:masterfrom
panbingkun:SPARK-51812

panbingkun commented Apr 16, 2025 •

edited

Loading

Uh oh!

panbingkun commented Apr 16, 2025

Uh oh!

LuciferYang left a comment

Uh oh!

summaryzb Apr 16, 2025

Uh oh!

panbingkun Apr 16, 2025

Uh oh!

LuciferYang commented Apr 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -606,7 +605,7 @@ object QueryExecution {
		session: SparkSession,

Conversation

panbingkun commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

panbingkun commented Apr 16, 2025

Uh oh!

LuciferYang left a comment

Choose a reason for hiding this comment

Uh oh!

summaryzb Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

panbingkun Apr 16, 2025

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Apr 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

panbingkun commented Apr 16, 2025 •

edited

Loading