apache / apache/datafusion-python

Pipeline FFI types: missing Protocols, ABCs, and capsule helpers

Aberta
#1,577 1 comentário 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
604
Forks
174
Merge médio
1d 7h
PRs com merge (30d)
4

Descrição

## Background

A v54 upstream coverage audit identified six FFI-pipeline gaps where DataFusion's FFI types are already imported on the Rust side but the Python surface is incomplete -- either missing a typed `Protocol` / `ABC` for users to implement against, or missing a `from_pycapsule` helper that the rest of the FFI surface uses. These are not v54-specific; they predate the v54 release. Filing this umbrella issue so the gaps are tracked together rather than disappearing into the audit report.

## Items

1. **`FFI_TableFunction`** -- no `TableFunctionExportable(Protocol)`. The duck-typed `hasattr` check at `user_defined.py:1161,1205` should be replaced by a typed protocol mirroring `TableProviderExportable`.

2. **`FFI_TableProvider`** -- no Python `TableProvider(ABC)` analogous to `CatalogProvider` / `SchemaProvider`. Users implementing a custom table provider in Python have no abstract base class to subclass; the Exportable protocol is the only entry point.

3. **`FFI_ExtensionOptions`** -- no `ExtensionOptionsExportable(Protocol)`. Currently consumed only at the `SessionConfig.with_extension` call site without a typed protocol describing the expected capsule export.

4. **`FFI_TaskContextProvider`** -- no Python `Protocol` and no example. Producer-only at `PySessionContext.__datafusion_task_context_provider__`; nothing imports one.

5. **`FFI_TableProviderFactory`** -- no `from_pycapsule` helper on `RustWrappedPyTableProviderFactory`. The capsule decode is inlined at `context.rs:728-749` rather than going through the `from_pycapsule!` macro the other FFI capsule importers use.

6. **`WindowUDF`** -- no Python `ABC` equivalent to `Accumulator` for native Python window UDFs. Users can register a Rust-defined window UDF over FFI but cannot define one in pure Python.

## Why deferred

Each item is small in isolation but they form a self-contained cleanup track. The audit grouped them so the work can be picked up together by a contributor focused on FFI ergonomics, rather than scattered across the v54 gap-closure PRs which focus on net-new surface area. Picking up any one item from this list is welcome.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Comece comparando as implementações existentes de TableProviderExportable, CatalogProvider, SchemaProvider, Accumulator e from_pycapsule! com os pontos de entrada nomeados em user_defined.py, context.rs, SessionConfig.with_extension e PySessionContext.__datafusion_task_context_provider__. Rastreie cada tipo FFI listado e sua importação Rust existente ou caminho de capsule. O trabalho estará concluído quando as seis interfaces tipadas ausentes, o helper e o exemplo estiverem cobertos de forma consistente e os testes relevantes passarem.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
python, rust
Domínio
api, developer-experience
Tipo de issue
Funcionalidade
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Pouca atividade
Clareza
Razoavelmente clara
Facilidade para iniciantes
48/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.