S3Queue auxiliary Zookeeper support#95203
Conversation
kssenii
added
the
can be tested
|
Workflow [PR], commit [e39170f] Summary: ❌
|
clickhouse-gh
bot
added
the
pr-improvement
|
|
||
| /// Use partition key for bucketing when bucketing_mode is PARTITION | ||
| /// This ensures files from the same partition always go to the same bucket | ||
| if (bucketing_mode == ObjectStorageQueueBucketingMode::PARTITION) | ||
| std::string key = path; | ||
| if (bucketing_mode == ObjectStorageQueueBucketingMode::PARTITION && hasPartitioningMode(partitioning_mode)) | ||
| { | ||
| auto partition_key = getPartitionKey(path, partitioning_mode, parser); | ||
| return sipHash64(partition_key) % buckets_num; | ||
| if (!partition_key.empty()) | ||
| key = std::move(partition_key); | ||
| } | ||
|
|
||
| /// Default hash the full file path | ||
| return sipHash64(path) % buckets_num; |
There was a problem hiding this comment.
Let's make it back to the old diff
| settings[ObjectStorageQueueSetting::mode] = table_metadata.mode; | ||
| settings[ObjectStorageQueueSetting::after_processing] = table_metadata.after_processing; | ||
| settings[ObjectStorageQueueSetting::keeper_path] = zk_path; | ||
| if (zookeeper_name == zkutil::DEFAULT_ZOOKEEPER_NAME) |
There was a problem hiding this comment.
Should we also check if it is empty here?
There was a problem hiding this comment.
Yes I saw this, the empty() one which was fixed but somehow it got through, I was gonna fix it. The thing here is that as @ianton-ru said, why we need to check for empty here if it has been already initialized?.
I was going to remove the empty condition from all.
Also since this was supposed to be fixed and it got through I'm going to review past comments and see if they are set.
|
Seems like we're ok 😄 Codex pulled the review comments from both PRs via gh and cross‑checked the current branch. Summary: Applied (matches current branch):
|
|
I've resolved a conflict in private sync, see |
lesandie
changed the title
@kssenii Seems like CH Inc sync test passed successfully |
b63af24
25.8.15 Backport of ClickHouse#95203: S3Queue auxiliary Zookeeper support
25.8.15 Backport of ClickHouse#95203: S3Queue auxiliary Zookeeper support
3 participants
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
S3Queue auxiliary Zookeeper support using
keeper_pathsetting from s3QueueThis will allow to do workload separation for heavy zk usage modes like unordered, by creating a zk/keeper specifically for some S3Queue tables and creating an S3queue specifying keeper_path with an auxiliary zookeeper like in a ReplicatedMergeTree table:
'auxiliary_zookeeper:/clickhouse/s3queue/my_s3queue_table'
Added integration tests and doc
Documentation entry for user-facing changes