apache / apache/datafusion-python

Make examples/ behave like examples: runnable, and consistent with the extension guide

オープン
#1,723 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
documentation enhancement
主要言語
Python
スター
604
フォーク
174
平均マージ
2日 22時間
マージ済み PR(30日)
5

説明

#1721 added `examples/distributed` as a worked example and, in doing so, exposed two structural problems with the older example trees that the new one throws into relief.

**The guide contradicts the code it points at.** `extension-guide/codecs.md` tells authors to encode durable metadata, then spends a paragraph explaining that our own examples do not. Readers copy examples; an example that contradicts the rule beside it is worse than no example. #1719 records the reference-implementation half of this as G9, which `examples/distributed/storage-library` now satisfies — but the crate a reader is most likely to open first still does the wrong thing.

**Two of the three trees have no runnable entry point.** `datafusion-ffi-example` is 13 pytest files and no script; `datafusion-ffi-query-planner-example` is one pytest file and no script. This is not only an aesthetic problem: `user-guide/data-sources.md:219` calls `datafusion-ffi-example` a **"user example"** and routes Python users to it from the user guide. They arrive at a test suite.

Investigation while reviewing #1721 found the first problem is smaller and differently shaped than the guide claims. Each item below was checked against the source, not inferred:

- `datafusion-ffi-query-planner-example` has **no** object registry at all — `grep -rn "HashMap\|OnceLock\|TOKEN" src/` is empty. It claims `node.is::()` narrowly and writes a fixed durable marker. The guide named it anyway. Corrected in #1721.
- `datafusion-ffi-example`'s **logical** codec parks providers in a process-global `HashMap`, and this is cheaply fixable: it claims `MemTable` narrowly (`logical_extension_codec.rs:148`), and a `MemTable` is fully describable as schema plus batches.
- `datafusion-ffi-example`'s **physical** codec parks plans, and this is *not* fixable here. It is forced by the greedy `ForeignExecutionPlan` claim, which is load-bearing for upstream defect apache/datafusion#25152 — that is G2 in #1719. Once you claim a node you cannot introspect, parking it is the only remaining option. The work is to make it uncopyable, not to remove it.

So the end state is one quarantined arm carrying an issue number and a delete-when condition, rather than a systemic contradiction.

**Decided while planning this:** the example trees stay in `examples/`. Relocating the conformance beds to an `integration-tests/` tree was considered and rejected — making them runnable resolves the "these aren't examples" complaint, and a move would churn roughly 40 doc, README, and agent-skill references, none of which any build step validates.

Sub-issues track the pieces. They are ordered: the prose rewrite describes the state the first two produce.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

まず extension-guide/codecs.md と user-guide/data-sources.md:219 から始め、続いて examples/ 配下の 3 つのツリーと datafusion-ffi-example/logical_extension_codec.rs:148 を調べます。既存の pytest スイートを実行して、現在のエントリーポイントを確認します。例が実行可能で、logical codec がプロセスグローバルな provider registry に依存せず、physical plan の隔離が明示的で、ガイドがコードと一致していれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python, rust
領域
developer-experience, documentation, testing
issue の種類
リファクタリング
難易度
4/5
見積もり時間
3〜5日
活発さ
活発
明瞭さ
おおむね明確
初心者へのやさしさ
48/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。