Skip to content

Unity catalog integration#76988

Merged
alesapin merged 49 commits intomasterfrom
unity_catalog
Mar 14, 2025
Merged

Unity catalog integration#76988
alesapin merged 49 commits intomasterfrom
unity_catalog

Conversation

@alesapin
Copy link
Member

@alesapin alesapin commented Feb 28, 2025

Changelog category (leave one):

  • Experimental Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add support for Unity Catalog for DeltaLake tables on top of AWS S3 and local filesystem.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

@clickhouse-gh
Copy link
Contributor

clickhouse-gh bot commented Feb 28, 2025 •
edited
Loading

Workflow [PR], commit [d296393]

@clickhouse-gh clickhouse-gh bot added the pr-experimental Experimental Feature label Feb 28, 2025
@alesapin alesapin changed the title Unity catalog Unity catalog integration Feb 28, 2025
@alesapin
Copy link
Member Author

alesapin commented Mar 1, 2025

hmm

#12 ERROR: denied: requested access to the resource is denied
------
 > pushing clickhouse/integration-test-with-unity-catalog:4577326297c8-aarch64 with docker:
------
ERROR: denied: requested access to the resource is denied

And what should I do?

kssenii
kssenii reviewed Mar 11, 2025
View reviewed changes
src/Databases/DataLake/DataLakeConstants.h

static constexpr auto DATABASE_ENGINE_NAME = "DataLakeCatalog";
static inline std::unordered_set<String> SETTINGS_TO_HIDE = {"catalog_credential", "auth_header"};
static constexpr std::string_view FILE_PATH_PREFIX = "file:/";
Copy link
Member

@kssenii kssenii Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why it is not file://?

Copy link
Member Author

@alesapin alesapin Mar 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because paths should start with /?

alesapin
alesapin commented Mar 12, 2025
View reviewed changes
alesapin
alesapin commented Mar 12, 2025
View reviewed changes
kssenii
kssenii reviewed Mar 13, 2025
View reviewed changes
@alesapin
Copy link
Member Author

alesapin commented Mar 14, 2025

I don't understand why flaky check failed. The idea of integration tests -- to run them in empty isolated env, why we run them in the same env in flaky check?

@alesapin alesapin added this pull request to the merge queue Mar 14, 2025
Merged via the queue into master with commit 35a3a37 Mar 14, 2025
120 of 129 checks passed
@alesapin alesapin deleted the unity_catalog branch March 14, 2025 10:06
@robot-clickhouse robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Mar 14, 2025
@alsugiliazova
Copy link
Contributor

alsugiliazova commented Mar 17, 2025

Do I understand correctly, by this PR you have changed database engine name from Iceberg to DataLakeCatalog?
If so, will it be backported to other versions?

@Blargian Blargian mentioned this pull request Mar 20, 2025
Closed
@alexey-milovidov alexey-milovidov mentioned this pull request Mar 21, 2025
Closed
76 tasks
@hodgesrm hodgesrm mentioned this pull request May 23, 2025
Closed
31 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kssenii kssenii kssenii approved these changes

Assignees

@kssenii kssenii

Labels

pr-experimental Experimental Feature pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@alesapin @alsugiliazova @kssenii @robot-clickhouse @svb-alt