Skip to content

Support references to unknown tables in remote table functions#83844

Merged
novikd merged 6 commits intomasterfrom
analyzer-fix-remote-function-view
Jul 22, 2025
Merged

Support references to unknown tables in remote table functions#83844
novikd merged 6 commits intomasterfrom
analyzer-fix-remote-function-view

Conversation

@novikd
Copy link
Member

@novikd novikd commented Jul 16, 2025

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Allow referencing any table in view(...) argument of remote table function with enabled analyzer. Fixes #78717. Fixes #79377.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

@clickhouse-gh
Copy link
Contributor

clickhouse-gh bot commented Jul 16, 2025

Workflow [PR], commit [ebf0db6]

Summary:

job_name test_name status info comment
Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, sequential) error
Upgrade check (amd_tsan) failure
S3_ERROR No such key thrown (see clickhouse-server.log or no_such_key_errors.txt) FAIL

@clickhouse-gh clickhouse-gh bot added the pr-bugfix Pull request with bugfix, not backported by default label Jul 16, 2025
@vdimir vdimir self-assigned this Jul 16, 2025
Copy link
Member

@vdimir vdimir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be a good idea to add a test for this case, an integration test, at the very least, should be feasible.

if (table_function_node->getTableFunctionName() == "view")
{
auto get_column_options = GetColumnsOptions(GetColumnsOptions::All).withExtendedObjects().withVirtuals();
auto column_names_and_types = distributed_storage_snapshot->getColumns(get_column_options);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How columns from distributed_storage_snapshot can be different from local analysis of table_function_node in case we have tables on all nodes? Is it the same? What if tables are present on all nodes, but has different structures?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Table structure can't differ on the shard because it's initialized from the structure request in getStructureOfRemoteTable. And we got in during table function analysis in QueryAnalyzer.

This code solves the problem that in remove(..., view(<>))expression view(<>) is not a resolved query tree and we can't resolve it on the initiator. It was resolved on the shard during the call in getStructureOfRemoteTable.

StorageID remote_storage_id = StorageID::createEmpty();
if (!remote_table_function_ptr)
remote_storage_id = StorageID{remote_database, remote_table};
StorageID remote_storage_id = StorageID{remote_database, remote_table};
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit controversial change, because for table functions it is always system.one.

@novikd novikd added this pull request to the merge queue Jul 22, 2025
@novikd novikd added the pr-must-backport Pull request should be backported intentionally. Use this label with great care! label Jul 22, 2025
Merged via the queue into master with commit 0f54691 Jul 22, 2025
238 of 243 checks passed
@novikd novikd deleted the analyzer-fix-remote-function-view branch July 22, 2025 14:57
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 22, 2025
@robot-ch-test-poll robot-ch-test-poll added pr-backports-created-cloud deprecated label, NOOP pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR labels Jul 22, 2025
robot-ch-test-poll4 added a commit that referenced this pull request Jul 22, 2025
Cherry pick #83844 to 24.8: Support references to unknown tables in remote table functions
robot-clickhouse added a commit that referenced this pull request Jul 22, 2025
robot-ch-test-poll4 added a commit that referenced this pull request Jul 22, 2025
Cherry pick #83844 to 25.3: Support references to unknown tables in remote table functions
robot-clickhouse added a commit that referenced this pull request Jul 22, 2025
robot-ch-test-poll4 added a commit that referenced this pull request Jul 22, 2025
Cherry pick #83844 to 25.4: Support references to unknown tables in remote table functions
robot-clickhouse added a commit that referenced this pull request Jul 22, 2025
robot-ch-test-poll4 added a commit that referenced this pull request Jul 22, 2025
Cherry pick #83844 to 25.5: Support references to unknown tables in remote table functions
robot-clickhouse added a commit that referenced this pull request Jul 22, 2025
robot-ch-test-poll4 added a commit that referenced this pull request Jul 22, 2025
Cherry pick #83844 to 25.6: Support references to unknown tables in remote table functions
robot-clickhouse added a commit that referenced this pull request Jul 22, 2025
robot-ch-test-poll4 added a commit that referenced this pull request Jul 22, 2025
Cherry pick #83844 to 25.7: Support references to unknown tables in remote table functions
robot-clickhouse added a commit that referenced this pull request Jul 22, 2025
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Jul 22, 2025
clickhouse-gh bot added a commit that referenced this pull request Jul 22, 2025
Backport #83844 to 25.7: Support references to unknown tables in remote table functions
clickhouse-gh bot added a commit that referenced this pull request Jul 22, 2025
Backport #83844 to 25.6: Support references to unknown tables in remote table functions
alexey-milovidov added a commit that referenced this pull request Jul 22, 2025
Backport #83844 to 25.5: Support references to unknown tables in remote table functions
novikd added a commit that referenced this pull request Jul 29, 2025
Backport #83844 to 25.3: Support references to unknown tables in remote table functions
ianton-ru pushed a commit to Altinity/ClickHouse that referenced this pull request Sep 2, 2025
zvonand added a commit to Altinity/ClickHouse that referenced this pull request Oct 10, 2025
Backport ClickHouse#83844 to 24.8: Support references to unknown tables in remote table functions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore pr-backports-created-cloud deprecated label, NOOP pr-bugfix Pull request with bugfix, not backported by default pr-must-backport Pull request should be backported intentionally. Use this label with great care! pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Issue with view() used with remoteSecure produces "UNKNOWN_TABLE" Error Enhanced remote table function for complex queries (more then one table)

4 participants