Skip to content

fix(python): guard DataFusion FFI export on datafusion major version#4142

Merged
ion-elgreco merged 2 commits intodelta-io:mainfrom
ethan-tyler:fix/4135-datafusion-ffi-guard
Jan 28, 2026
Merged

fix(python): guard DataFusion FFI export on datafusion major version#4142
ion-elgreco merged 2 commits intodelta-io:mainfrom
ethan-tyler:fix/4135-datafusion-ffi-guard

Conversation

@ethan-tyler
Copy link
Copy Markdown
Collaborator

Description

  • Add a runtime version check in datafusion_table_provider to prevent FFI ABI mismatch segfaults
  • Block capsule export when installed datafusion major != 52
  • Provide actionable error text with QueryBuilder workaround

Changes:

  • lib.rs: add REQUIRED_DATAFUSION_PY_MAJOR, datafusion_python_version(), guard at method start
  • test_datafusion.py: add incompatible version and not installed tests

Note: This guard is a temporary safety net to prevent segfaults until DataFusion 52 Python wheels are available on PyPI. Once wheels land, users can install datafusion==52.* and use SessionContext registration normally.

Related Issue(s)

Documentation

@github-actions github-actions Bot added the binding/python Issues for the Python package label Jan 28, 2026
@rtyler rtyler enabled auto-merge (rebase) January 28, 2026 18:17
@ion-elgreco ion-elgreco disabled auto-merge January 28, 2026 18:18
@codecov
Copy link
Copy Markdown

codecov Bot commented Jan 28, 2026

Codecov Report

❌ Patch coverage is 0% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.17%. Comparing base (6ee66e2) to head (f2d76dc).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
python/src/lib.rs 0.00% 25 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4142      +/-   ##
==========================================
- Coverage   76.23%   76.17%   -0.06%     
==========================================
  Files         164      164              
  Lines       45468    45471       +3     
  Branches    45468    45471       +3     
==========================================
- Hits        34661    34638      -23     
- Misses       9171     9198      +27     
+ Partials     1636     1635       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Collaborator

@ion-elgreco ion-elgreco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe it should be only equal to a major version but greater than or equal to

@rtyler
Copy link
Copy Markdown
Member

rtyler commented Jan 28, 2026

@ion-elgreco if this version of the python package were used in the future with a datafusion-ffi of 53 wouldn't there still likely be a SIGSEGV due to ABI incompatibilities?

Basically as I understand the problem here, this version number will have to change as newer major releases are built into our wheels

@ion-elgreco
Copy link
Copy Markdown
Collaborator

@ion-elgreco if this version of the python package were used in the future with a datafusion-ffi of 53 wouldn't there still likely be a SIGSEGV due to ABI incompatibilities?

Basically as I understand the problem here, this version number will have to change as newer major releases are built into our wheels

Only if the ffi interface changes but that's assuming we will have breaking changes every major release of datafusion-ffi. But before it was working for multiple major versions

@rtyler
Copy link
Copy Markdown
Member

rtyler commented Jan 28, 2026

@ion-elgreco in the future we can update the check with the new major versions, but I think we should be strict until newer versions are proven to work. The consequences of failing is a complete segfault which is about as disastrous of an outcome as it gets

@ion-elgreco
Copy link
Copy Markdown
Collaborator

@rtyler alright that's fair

@ion-elgreco ion-elgreco merged commit 5121983 into delta-io:main Jan 28, 2026
28 of 29 checks passed
@ethan-tyler ethan-tyler deleted the fix/4135-datafusion-ffi-guard branch January 29, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

binding/python Issues for the Python package

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants