[GLUTEN-6067][CH] [Part 3-2] Basic support for Native Write in Spark 3.5#6586
Merged
baibaichen merged 5 commits intoapache:mainfrom Jul 26, 2024
Merged
[GLUTEN-6067][CH] [Part 3-2] Basic support for Native Write in Spark 3.5#6586baibaichen merged 5 commits intoapache:mainfrom
baibaichen merged 5 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI |
1 similar comment
|
Run Gluten Clickhouse CI |
…tiveBlock::toColumnarBatch() to return ColumnarBatch 2. Extract a mew function SerializedPlanParser::buildPipeline, which used in the follow up PRs 3. Refactor File Wrapper, extract create_output_format_file for later use 4. Add GLUTEN_SOURCE_DIR, so that gtest can read java resource 5. Add SubstraitParserUtils.h, so that we can remove parseJson 6. Many litter refactor
084807c to
f78ffb3
Compare
|
Run Gluten Clickhouse CI |
Contributor
Author
|
we run |
17 tasks
zzcclp
approved these changes
Jul 26, 2024
taiyang-li
reviewed
Jul 26, 2024
| { | ||
| auto plan_ptr = std::make_unique<substrait::Plan>(); | ||
| auto success = plan_ptr->ParseFromString(plan); | ||
| substrait::Plan sPlan; |
Contributor
There was a problem hiding this comment.
use the naming rule of clickhouse
taiyang-li
reviewed
Jul 26, 2024
| /// 1. global level resources like global_context/shared_context, notice that they can only be initialized once in process lifetime | ||
| /// 2. session level resources like settings/configs, they can be initialized multiple times following the lifetime of executor/driver | ||
| static void init(const std::string & plan); | ||
| static void init(const std::string_view plan); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
(Fixes: #6067)
This PR implement baisc support for native write in spark 3.5. I first refactor codes, so that we can add sink transform after parsing substrait plan.
The main idea is using
PartitionedSink, and the core of this pr is how to compute partition value, see following codesHow was this patch tested?
Using existed UTs