apache / apache/datafusion-python
Add an agent skill for datafusion-python extension authors
- 主要言語
- Python
- スター
- 604
- フォーク
- 174
- 平均マージ
- 1日 7時間
- マージ済み PR(30日)
- 4
説明
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Writing a datafusion-python extension library means getting a handful of non-obvious lifetime and identity rules right, and there is currently no guidance an agent working in a *downstream* repo can find. The rules are spread across `docs/source/contributor-guide/ffi.md`, the docstrings on `SessionContext.with_extensions` / `with_logical_extension_codec`, `.ai/skills/ffi-capsule-protocol/SKILL.md`, and the two example crates. Each of those is written for someone already inside this repository.
The people who need it are not. Ballista and datafusion-distributed each build a separate cdylib plus Python glue against this project's FFI protocol, and they hit the same set of traps that #1679 was opened to fix: components bound to an intermediate context that is later collected, `TaskContextProvider went out of scope over FFI boundary`, codec ids that do not survive a process boundary, and `write_parquet` failing across the FFI codec because it has no file-format hooks. Every one of those is knowable in advance and none of them is discoverable from where those authors are standing.
Raised by @ntjohnson1 in review of #1679: *"I wonder if it makes sense to have a todo for some datafusion python extension skill/s ... I do suspect it will require some guidance to make sure they are doing it safely."*
**Describe the solution you'd like**
An agent skill for extension *authors*, plus a contributor-side audit skill that keeps it from rotting.
*Location.* A new `skills/datafusion_python_extension/SKILL.md`, sibling to the existing `skills/datafusion_python/SKILL.md`. Not `.ai/skills/` — that tier is for agents working on this repository, and this audience is working on a different one. `.ai/skills/ffi-capsule-protocol/SKILL.md` stays where it is as the review trigger for changes to the capsule getters themselves, and the two cross-reference rather than duplicate.
*Open question this issue needs to settle: how does a downstream agent get the file?* `skills/` is not currently included in the wheel, so an agent in a repo that depends on `datafusion` cannot see it. Options, roughly in order of how well they match "keeping it up to date shouldn't be too bad":
1. Ship `skills/` as package data in the wheel. Downstream gets the current version with the dependency they already have, and it tracks the release they are actually building against.
2. Publish to agentskills.io, the standard `CLAUDE.md` already points at, so `find-skills` discovers it.
3. Document that downstream repos should vendor a copy. Cheapest to do, fastest to go stale.
*Content.* Drawn from the traps that #1672 and its three split PRs actually surfaced, not invented advice:
- The `__datafusion_session_extension__` bundle protocol: create fresh components on every call, never cache bound components, never retain the context that was passed in.
- Take the task-context provider off the supplied context. Never construct a `SessionContext` inside an extension library. (Link `ffi-capsule-protocol`; do not restate it.)
- Codec identity: the class-derived id, `__datafusion_codec_id__` for pinning one, and `anon:` for a bare `PyCapsule`. A plan that has to decode in another process requires a declared id — this is the rule a distributed engine hits first.
- Ownership: the context returned from `with_extensions` is the strong owner. A `DataFrame` does not keep it alive. `TaskContextProvider went out of scope over FFI boundary` is the symptom; holding the context is the cure.
- Prefer `with_extensions` to hand-chaining `with_logical_extension_codec` / `with_physical_extension_codec` / `set_query_planner`, and say why.
- Known gap: `FFI_LogicalExtensionCodec` has no vtable entries for file formats, so `df.write_csv` / `write_parquet` / `write_json` fail across the boundary. Compose the FFI codec with a `DefaultLogicalExtensionCodec` fallback on `try_encode_file_format` / `try_decode_file_format`.
*Sequencing.* This should land after the codec-id gap in `with_extensions` is resolved. `SessionExtensionComponents` currently has no way to declare an id for a codec handed over as a bare capsule, and `with_extensions` takes no `codec_id=`, so the only workaround is a Python shim that declares `__datafusion_codec_id__`. Writing the skill now would document that workaround as the intended design and require a rewrite when the API gains a way to express it.
*Keeping it current.* An `.ai/skills/audit-extension-skill/` contributor skill following the existing `.ai/skills/audit-skill-md/` pattern: diff the skill's claims against the two example crates and the current public API, and report drift. This is the part worth automating, because the failure mode is the skill silently describing a protocol that has since moved.
**Describe alternatives you've considered**
*Extend `.ai/skills/ffi-capsule-protocol` instead of adding a skill.* Wrong audience and wrong trigger. That skill fires when someone touches a `__datafusion_*__` getter in this repository. An extension author is not touching those; they are implementing against them, and they need a build-it-from-scratch narrative rather than a review checklist.
*Extend `docs/source/contributor-guide/ffi.md` instead.* Already the reference material, and #1679 adds substantially to it. But it is organized around the protocol rather than around the task, and it lives at a URL a downstream agent has no reason to fetch. The skill should point *at* `ffi.md` for depth rather than replace it.
*Assert in CI that the skill can regenerate the three-library example.* This was the original suggestion and it is the right instinct — the example is the best available proof the guidance is sufficient — but it does not survive contact with a workflow. Generated output varies between runs, so a byte-comparison fails on noise and a fuzzy comparison asserts nothing. The deterministic half is the audit skill above. The full generate-from-skill-and-run-the-tests exercise is real value in the wrong shape for CI; it belongs on the release checklist as a manual step.
**Additional context**
Reference implementations the skill should point at, both exercised in CI via `.github/workflows/test.yml`:
- `examples/datafusion-ffi-example/` — table providers, UDFs, catalog providers, optimizer rules, extension codecs.
- `examples/datafusion-ffi-query-planner-example/` — query planner, and `MyPlannerExtension` in `src/extension.rs` as a complete Rust implementation of `__datafusion_session_extension__`.
Follows up on #1679 (part 3 of the #1672 split).
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
Start with skills/datafusion_python/SKILL.md, .ai/skills/audit-skill-md/, docs/source/contributor-guide/ffi.md, the two example crates, and the public with_extensions APIs. Resolve how downstream agents receive the skill and wait for the codec-id gap to land, then define the author skill and audit skill against the documented traps and current examples. Done means the guidance has an agreed delivery path and a repeatable drift check.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python, rust
- 領域
- developer-experience, documentation
- issue の種類
- ドキュメント
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 活発さ
- 活発
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100