Workflow [PR], commit [e0d5e8d]

Summary: ❌

@scanhex12,

It should be implemented in the following way, not related to row groups or data lakes:

• define a setting, desired job size in bytes;
• add two new fields to the protocol for distributed files processing: bucket num and bucket size;
• when a file is larger than this threshold, we pass the bucket number from 0 to N-1 for processing;
• the bucket number defines a deterministic split of the file into smaller parts for processing;
• however, this split will depend on the actual structure of the file, and sometimes certain buckets could be empty or the distribution can be uneven;
• this will work for Parquet, ORC, and could be extended to any other formats.

The coordinator knows file sizes and, therefore, knows how many buckets each file will need. Then distributes the work for each bucket in each file. We can say "virtual buckets" to better understand the concept, because the actual split of the file and the way how it is split is abstracted.

Latest times measurements on integration test with parquet v3 (first time is bucket-level splitting):

Check times 0.5894033908843994, 0.6704092025756836