apache / apache/datafusion

`PhysicalExtensionCodec` should take a `PhysicalPlanDecodeContext` instead of a `TaskContext`

Open
#25,089 2 comments 0 reactions 1 assignee Claimed by @namanjain24-sudo View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The current `PhysicalExtensionCodec` trait looks like:
```
pub trait PhysicalExtensionCodec: Debug + Send + Sync + Any {
fn try_decode(
&self,
buf: &[u8],
inputs: &[Arc],
ctx: &TaskContext,
proto_converter: &dyn PhysicalProtoConverterExtension,
) -> Result>;

...other methods
}
```

which is unfortunate because it will cause decoding of extension plans to fail if they contain scalar subquery expressions. The `ScalarSubqueryResults` are stages in the `PhysicalPlanDecodeContext` when decoding a `ScalarSubqueryExec` and are required when decoding a `ScalarSubqueryExpr`

### Describe the solution you'd like

Two options I can see:

1. Breaking change: Change the method signature for `PhysicalExtensionCodec::try_decode`:

```
pub trait PhysicalExtensionCodec: Debug + Send + Sync + Any {
fn try_decode(
&self,
buf: &[u8],
inputs: &[Arc],
ctx: PhysicalPlanDecodeContext<'_>,
proto_converter: &dyn PhysicalProtoConverterExtension,
) -> Result>;

...other methods
}
```

Pretty trivial to adapt to but is still a breaking change.

2. Non-breaking change: Add a new method `PhysicalExtensionCodec:try_decode_with_ctx` which has a default implementation falling back to the existing method:
```
pub trait PhysicalExtensionCodec: Debug + Send + Sync + Any {
fn try_decode(
&self,
buf: &[u8],
inputs: &[Arc],
ctx: &TaskContext,
proto_converter: &dyn PhysicalProtoConverterExtension,
) -> Result>;

fn try_decode_with_ctx(
&self,
buf: &[u8],
inputs: &[Arc],
ctx: PhysicalPlanDecodeContext<'_>,
proto_converter: &dyn PhysicalProtoConverterExtension,
) -> Result> {
self.try_decode(buf, inputs, ctx.task_ctx(), proto_converter)
}

...other methods
}
```

### Describe alternatives you've considered

I *think* you can hack around this with `PhysicalProtoConverterExtension` somehow but it seems pretty convoluted

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.