apache / apache/datafusion

Serializing custom `TableSource` implementations fails

Open
#16,749 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

`LogicalExtensionCodec` allows providing a custom serialization strategy for a `TableProvider`, but the calling code always expects a `DefaultTableSource` to unwrap:

https://github.com/apache/datafusion/blob/8d7b11b1cdf04aa9e7c402b5c97e45240e6e71a0/datafusion/proto/src/logical_plan/mod.rs#L1004

Which calls `source_as_provider`:

https://github.com/apache/datafusion/blob/8d7b11b1cdf04aa9e7c402b5c97e45240e6e71a0/datafusion/catalog/src/default_table_source.rs#L99-L110

It would be better if `LogicalExtensionCodec` had `encode_table_source` instead of `encode_table_provider`.

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/proto/src/logical_plan/mod.rs around line 1004 and trace the LogicalExtensionCodec call into datafusion/catalog/src/default_table_source.rs lines 99-110. Confirm how custom TableSource implementations reach source_as_provider, then verify that serialization no longer requires a DefaultTableSource unwrap and that the relevant proto or catalog tests cover the custom case.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.