Could you please split this test into separate ones? Flaky check fails because a single test runs for too long https://s3.amazonaws.com/clickhouse-test-reports/PRs/81040/27bf90da21b8557c55b73a9ae98cdfc01707b626//integration_tests_amd_asan_flaky/job.log
Also please avoid force-push from now on, as I've fixed private synchronization conflicts and a force push will reset my fixes.

@ianton-ru, hi, could you please clarify why do we need this check? As we use persistent nodes for processing/buckets nodes, this error should not happen, otherwise it would mean duplicated data which must not happen and this retry could just hide some bugs

Scenario with multiple nodes:

• two hive partition, part1 with file file11 and part2 with file file21.
• records in Keeper have version 1
• node1 processes files file11 and file21
• meanwhile new file file12 is added in part1
• node2 processes file file12
• node1 complete processing, starts to create records list. Extract versions for parts, both have version 1
• node2 complete processing, starts to create records list, with version1 too
• node2 commits records without errors, record in Keeper for part1 gets version 2
• node1 tries to commit records as single transaction, gets error Bad Version for part1
  Without hive partitions node1 does not need to write something after that, because node2 wrote more actual information about last processing file.
  But with hive partitions node1 still need to write last processing file for part2.

So you mean node1 was processing file /part1/file11 concurrently with node2 processing file /part1/file12?
But if file11 and file12 share the same processed_node_path, then they are in the same bucket, and a single bucket can only be processed by a single processor, not allowing any concurrent processing (persistent bucket lock must maintain this invariant). Then the situation you describe should not happen.

Hmm. Looks like ClickHouse locks buckets only when number of buckets is greater than 1 (when useBucketsForProcessing returns true).
Is configuration with multiple nodes and single bucket valid?

Is configuration with multiple nodes and single bucket valid?

Not valid as it can lead to inconsistent processing (in some failure scenarios we can end up with files which will never be processed) and retries will not work properly. Currently if processing_threads_num is >1 (it will almost always be so as we take default from the number of cpu cores) and buckets is disabled, then buckets are anyway enforced automatically as well. Moreover, in cloud we do not allow to create tables without bucket-based processing:

Should be handled here https://github.com/ClickHouse/clickhouse-private/blob/d442af4b207906e5c3600e35eb82742a1d0f2d97/src/Storages/ObjectStorageQueue/ObjectStorageQueueTableMetadata.h#L102-L106

Oh. I use "buckets=1" in tests, getBucketsNum returns 1, because buckets is not zero.

It's a gray zone. useBucketsForProcessing checks that buckets_num>1.

useBucketsForProcessing checks that buckets_num>1

Yes, but ObjectStorageQueueMetadata sets buckets_num as table_metadata->getBucketsNum, which returns what I've sent in the above message https://github.com/ClickHouse/clickhouse-private/blob/d442af4b207906e5c3600e35eb82742a1d0f2d97/src/Storages/ObjectStorageQueue/ObjectStorageQueueTableMetadata.h#L102-L106

I use "buckets=1" in tests,

In this case yes, it will not work, this is an omission which needs to be fixed. But by default buckets = 0, so if user does not touch this setting and only changes processing_threads_num, then buckets must be correctly set to the value of processing_threads_num

Removed here: #93843