apache / apache/datafusion-python
Report which extension codec handled each node of a serialized plan
- Lingua principale
- Python
- Stelle
- 604
- Fork
- 174
- Merge medio
- 1g 7h
- PR unite (30g)
- 4
Descrizione
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Extension codecs compose as of #1678: a session holds a chain of them, and encoding walks the chain in install order until one claims an object. With several libraries installed there is currently no way to ask which codec handled a given node. Raised in https://github.com/apache/datafusion-python/pull/1678#pullrequestreview-4940081598.
Part of this is answered already. `SessionContext.logical_extension_codec_ids()` and `physical_extension_codec_ids()` list what is installed, in install order, and those ids are what a payload carries — so a decode failure names the codec that wrote the bytes and lists what the session actually has. What is missing is per-call attribution: which codec handled which node of a particular plan. Today the only way to find out is to check a codec's own call counters before and after, which requires the codec to expose them and tells you nothing about which node was involved.
**Describe the solution you'd like**
Surface the winning codec per node, most naturally in `EXPLAIN` output for a plan that has been serialized, or failing that as an inspection call that reports the encode decisions made for a given plan.
**Describe alternatives you've considered**
Leaving it to the decode error, which already names the responsible codec. That covers the case where something went wrong but not the case where someone is trying to understand a working setup, which is when a multi-library chain is most confusing.
Logging each claim at debug level. Cheap to add and much weaker: it is per-session rather than per-plan, and it puts the burden of correlating lines with nodes on the reader.
**Additional context**
Deliberately left out of #1678: recording the winner means threading it through plan formatting, which is a change to how plans are displayed rather than to how codecs compose.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Start with SessionContext.logical_extension_codec_ids() and physical_extension_codec_ids(), then read the extension codec chain described in #1678. Determine how a serialized plan is represented for EXPLAIN and define how the winning codec can be attributed to each node; done means a plan-specific inspection or EXPLAIN result identifies the codec handling every serialized node.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- backend-api-design, data-engineering
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Attiva
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100