DataFusion integration tests fail after release of datafusion 52
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 91
Description
It looks like the release of DataFusion 52.0.0 causes the following tests to segfault. This is experimental functionality that we haven't explored...we probably need to check the major version of DataFusion before attempting an import via the dunder method since that ABI is not yet stable.
```python
def test_udf_sedonadb_registry_function_to_datafusion(con):
datafusion = pytest.importorskip("datafusion")
udf_impl = udf.arrow_udf(pa.binary(), [udf.STRING, udf.NUMERIC])(some_udf)
# Register with our session
con.register_udf(udf_impl)
# Create a datafusion session, fetch our udf and register with the other session
datafusion_ctx = datafusion.SessionContext()
datafusion_ctx.register_udf(
datafusion.ScalarUDF.from_pycapsule(con._impl.scalar_udf("some_udf"))
)
# Can't quite use to_pandas() because there is a schema/batch nullability mismatch
batches = datafusion_ctx.sql("SELECT some_udf('abcd', 123) as col").collect()
assert len(batches) == 1
pd.testing.assert_frame_equal(
batches[0].to_pandas(),
pd.DataFrame({"col": [b"abcd / 123"]}),
)
def test_udf_sedonadb_to_datafusion():
datafusion = pytest.importorskip("datafusion")
udf_impl = udf.arrow_udf(pa.binary(), [udf.STRING, udf.NUMERIC])(some_udf)
# Create a datafusion session, register udf_impl directly
datafusion_ctx = datafusion.SessionContext()
datafusion_ctx.register_udf(datafusion.ScalarUDF.from_pycapsule(udf_impl))
# Can't quite use to_pandas() because there is a schema/batch nullability mismatch
batches = datafusion_ctx.sql("SELECT some_udf('abcd', 123) as col").collect()
assert len(batches) == 1
pd.testing.assert_frame_equal(
batches[0].to_pandas(),
pd.DataFrame({"col": [b"abcd / 123"]}),
)
```
Contributor guide
Research direction
Start by running the two named integration tests with DataFusion 52.0.0 and inspect the DataFusion import and from_pycapsule calls shown in the issue. Check how the DataFusion major version is detected before the dunder-based import path. Done means the tests no longer segfault and the supported integration behavior is preserved or safely avoided for incompatible versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- database, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100