apache / apache/sedona-db

DataFusion integration tests fail after release of datafusion 52

Open
#655 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 5h
Merged PRs (30d)
91

Description

It looks like the release of DataFusion 52.0.0 causes the following tests to segfault. This is experimental functionality that we haven't explored...we probably need to check the major version of DataFusion before attempting an import via the dunder method since that ABI is not yet stable.

```python

def test_udf_sedonadb_registry_function_to_datafusion(con):
datafusion = pytest.importorskip("datafusion")
udf_impl = udf.arrow_udf(pa.binary(), [udf.STRING, udf.NUMERIC])(some_udf)

# Register with our session
con.register_udf(udf_impl)

# Create a datafusion session, fetch our udf and register with the other session
datafusion_ctx = datafusion.SessionContext()
datafusion_ctx.register_udf(
datafusion.ScalarUDF.from_pycapsule(con._impl.scalar_udf("some_udf"))
)

# Can't quite use to_pandas() because there is a schema/batch nullability mismatch
batches = datafusion_ctx.sql("SELECT some_udf('abcd', 123) as col").collect()
assert len(batches) == 1
pd.testing.assert_frame_equal(
batches[0].to_pandas(),
pd.DataFrame({"col": [b"abcd / 123"]}),
)

def test_udf_sedonadb_to_datafusion():
datafusion = pytest.importorskip("datafusion")
udf_impl = udf.arrow_udf(pa.binary(), [udf.STRING, udf.NUMERIC])(some_udf)

# Create a datafusion session, register udf_impl directly
datafusion_ctx = datafusion.SessionContext()
datafusion_ctx.register_udf(datafusion.ScalarUDF.from_pycapsule(udf_impl))

# Can't quite use to_pandas() because there is a schema/batch nullability mismatch
batches = datafusion_ctx.sql("SELECT some_udf('abcd', 123) as col").collect()
assert len(batches) == 1
pd.testing.assert_frame_equal(
batches[0].to_pandas(),
pd.DataFrame({"col": [b"abcd / 123"]}),
)
```

Contributor guide

Open the contributing guide

Research direction

Start by running the two named integration tests with DataFusion 52.0.0 and inspect the DataFusion import and from_pycapsule calls shown in the issue. Check how the DataFusion major version is detected before the dunder-based import path. Done means the tests no longer segfault and the supported integration behavior is preserved or safely avoided for incompatible versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
database, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.