apache / apache/datafusion

make_array: relax element-type equality to accept inputs differing only in nested-field nullability

オープン
#22,366 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Rust
スター
9.3k
フォーク
2.4k
平均マージ
3日 11時間
マージ済み PR(30日)
360

説明

## Summary

`make_array` (in `datafusion-functions-nested`) panics when called with arrays whose element types share the same shape but differ in nested-field nullability. Spark, Postgres, and `arrow::compute::concat` all accept this and widen `nullable` to `true` in the result type. DataFusion's `make_array_inner` is stricter, which propagates up to any caller that builds `array(...)` over heterogeneously-produced child expressions.

## Repro symptom

Real-world surfacing in [apache/datafusion-comet](https://github.com/apache/datafusion-comet) on a Delta Lake CDF write that builds `array(struct(id, b, _change_type=lit(\"delete\")), struct(id, b, _change_type=col(...)))` — one arm's `_change_type` is `Utf8` non-nullable (from a literal), another is `Utf8` nullable:

```
panicked at arrow-data-58.2.0/src/transform/mod.rs:422:
assertion `left == right` failed: Arrays with inconsistent types passed to MutableArrayData
left: Struct([Field { name: \"id\", data_type: Int64, nullable: true },
Field { name: \"b\", data_type: Int32 },
Field { name: \"_change_type\", data_type: Utf8 }])
right: Struct([Field { name: \"id\", data_type: Int64, nullable: true },
Field { name: \"b\", data_type: Int32 },
Field { name: \"_change_type\", data_type: Utf8, nullable: true }])
```

Stack: `make_array_inner` → `MutableArrayData::with_capacities`.

## Proposal

`make_array` should accept element types that are equal under nullability-widening (recursively, for nested structs/lists/maps). Concretely:

- Compute the merged element type by walking each child's `DataType` and OR-ing the `nullable` flag at every level (this is essentially `Field::try_merge` minus the type-promotion arm).
- Cast each child to the merged type before handing to `MutableArrayData`.
- Return `ArrayType` with `containsNull = true` if any merge raised a nullability flag.

This matches what `coerce_types`-style coercion does elsewhere in the planner, but applied at execution time when input arrays still disagree (the planner can't always normalize, e.g. when the array is built from disjoint sources like Delta CDF struct literals).

## Why this matters

It blocks native execution of any plan that produces struct elements from multiple sources (CDF writes, UNION ALL inside an `array()`, manually-constructed plans bypassing TypeCoercion). Workaround today: callers must insert explicit casts upstream, or fall back to a non-DataFusion evaluator — both of which lose perf.

## Related caller-side mitigation (for context)

Comet just landed a serde-side decline in [4cb9b4dc](https://github.com/apache/datafusion-comet/commit/) that falls back to Spark's JVM evaluator when `CreateArray`'s children have different `DataType`s. That fix is conservative but loses native execution. Upstreaming the relaxation here would let downstream projects keep native execution and would help any other Arrow-based engine hitting the same shape.

I can put up a PR if the approach lands well.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

datafusion-functions-nested の実装にある make_array_inner から開始し、MutableArrayData::with_capacities への呼び出しを追跡します。既存の型処理を、説明されている coerce_types スタイルの動作と比較し、ネストされた nullability の違いが広げられること、子が panic なしで受け入れられること、そして結果の配列がマージされた nullability を報告することを確認します。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
rust
領域
data-engineering
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
静か
明瞭さ
明確に書かれている
初心者へのやさしさ
55/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。