apache / apache/datafusion-python
Report which extension codec handled each node of a serialized plan
- Ngôn ngữ chính
- Python
- Star
- 604
- Fork
- 174
- Merge trung bình
- 1 ngày 7 giờ
- Pull request đã merge (30 ngày)
- 4
Mô tả
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Extension codecs compose as of #1678: a session holds a chain of them, and encoding walks the chain in install order until one claims an object. With several libraries installed there is currently no way to ask which codec handled a given node. Raised in https://github.com/apache/datafusion-python/pull/1678#pullrequestreview-4940081598.
Part of this is answered already. `SessionContext.logical_extension_codec_ids()` and `physical_extension_codec_ids()` list what is installed, in install order, and those ids are what a payload carries — so a decode failure names the codec that wrote the bytes and lists what the session actually has. What is missing is per-call attribution: which codec handled which node of a particular plan. Today the only way to find out is to check a codec's own call counters before and after, which requires the codec to expose them and tells you nothing about which node was involved.
**Describe the solution you'd like**
Surface the winning codec per node, most naturally in `EXPLAIN` output for a plan that has been serialized, or failing that as an inspection call that reports the encode decisions made for a given plan.
**Describe alternatives you've considered**
Leaving it to the decode error, which already names the responsible codec. That covers the case where something went wrong but not the case where someone is trying to understand a working setup, which is when a multi-library chain is most confusing.
Logging each claim at debug level. Cheap to add and much weaker: it is per-session rather than per-plan, and it puts the burden of correlating lines with nodes on the reader.
**Additional context**
Deliberately left out of #1678: recording the winner means threading it through plan formatting, which is a change to how plans are displayed rather than to how codecs compose.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
Start with SessionContext.logical_extension_codec_ids() and physical_extension_codec_ids(), then read the extension codec chain described in #1678. Determine how a serialized plan is represented for EXPLAIN and define how the winning codec can be attributed to each node; done means a plan-specific inspection or EXPLAIN result identifies the codec handling every serialized node.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- backend-api-design, data-engineering
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 35/100