apache / apache/datafusion-comet

Relax the arrow version requirement in comet-udf-sdk

Open
#5,253 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

Follow-up from review of #4459 ([thread](https://github.com/apache/datafusion-comet/pull/4459#discussion_r3712387025)).

`comet-udf-sdk` inherits `arrow = { workspace = true }`, currently `58.4.0`. Because the SDK is compiled into the user's cdylib, Cargo has to unify its `arrow` requirement with the user's, so a UDF author is effectively pinned to `arrow` 58.x whatever their own project uses.

Nothing about the ABI requires that. Only `FFI_ArrowArray` and `FFI_ArrowSchema` cross the boundary, and those are `#[repr(C)]` renderings of the Arrow C Data Interface, stable across arrow-rs versions. As @paleolimbot noted, the requirement could in principle be relaxed back to whenever those types were added.

Not urgent: the crate is `publish = false` and users depend on it by git, so there are no external consumers to unblock yet. It becomes the blocking constraint if and when the SDK is published.

Things to work out:

- A relaxed range on one workspace member needs an override rather than `workspace = true`, and it should not perturb how the rest of the workspace resolves.
- The floor needs establishing by actually compiling against it, not by reading changelogs.
- CI would want a job building the SDK against the floor version, otherwise the range is a claim nothing tests.

Contributor guide

Open the contributing guide

Research direction

Start in the comet-udf-sdk Cargo configuration and inspect how its workspace Arrow dependency is resolved. Compile the SDK against candidate floor versions, then check the CI configuration for a job that exercises the floor. Done means the SDK has a tested relaxed requirement without changing resolution for the rest of the workspace.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, ci-cd
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.