apache / apache/datafusion

make_array / array_agg drop inner-Field metadata when constructing List<T>

Open
#21,982 3 comments 0 reactions 1 assignee Claimed by @CuteChuanChuan View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Several built-in UDFs and UDAFs that wrap an input column/field into a composite output type (`List`, `Struct`, `Map`, …) drop the input field's metadata when constructing the output's inner Field.

In practice this matters most for Arrow extension types (`ARROW:extension:name` / `ARROW:extension:metadata` — e.g. `arrow.json`, `arrow.uuid`), because SQL-constructed lists/structs silently lose extension-type identity. Any downstream operation that compares the produced `DataType` against a type carrying inner-field metadata (union, aggregate merging, IPC roundtrip) will see them as different types.

### MRE

Using the existing `with_metadata` and `arrow_field` SQL functions:

```sql
-- make_array: inner field's metadata is dropped.
SELECT arrow_field(make_array(with_metadata('a', 'key', 'value')));
```

```
+-----------------------------------------------------------------------------+
| arrow_field(make_array(with_metadata(Utf8("a"),Utf8("key"),Utf8("value")))) |
+-----------------------------------------------------------------------------+
| {name: lit, data_type: List(Utf8), nullable: false, metadata: {}} |
+-----------------------------------------------------------------------------+
```

`array_agg` suffers from the same issue.

### Root cause

Both functions implement only the type-only `return_type` hook and never look at input fields, so they have no way to read source metadata:

- `datafusion/functions-nested/src/make_array.rs:97-106` — `return_type` returns `DataType::new_list(element_type, true)`.
- `datafusion/functions-aggregate/src/array_agg.rs:107-112` — `return_type` returns `DataType::List(Arc::new(Field::new_list_field(arg_types[0].clone(), true)))`.

The runtime paths construct fresh fields with no metadata as well:

- `array_array` in `make_array.rs:237` builds `Arc::new(Field::new(field_name, data_type, true))`.
- The various `array_agg` accumulators (`array_agg.rs:663`, `:734`, etc.) build `Arc::new(Field::new_list_field(self.datatype.clone(), true))`.

Both UDF (`return_field_from_args`) and UDAF (`return_field`) hooks accept input `FieldRef`s and could propagate metadata, but neither function overrides them.

### Audit: other built-in UDFs/UDAFs with the same pattern

The same shape — only `return_type` is overridden, runtime synthesizes a fresh inner Field — was found in:

| Function | File:line of `return_type` | Notes |
| --- | --- | --- |
| `make_array` | `functions-nested/src/make_array.rs:97` | known; runtime at L237 builds `Field::new(name, dt, true)` |
| `array_agg` | `functions-aggregate/src/array_agg.rs:107` | known; runtime accumulators (L663, L734) use `Field::new_list_field(self.datatype, true)` |
| `map` | `functions-nested/src/map.rs:381` | constructs fresh `Field::new("key", …)` / `Field::new("value", …)` at L384–L393 |
| `range` / `generate_series` | `functions-nested/src/range.rs:226` | `Field::new_list_field(element_type, true)` at L234, L238, L243 (low-impact since args are usually scalars) |
| `arrays_zip` | `functions-nested/src/arrays_zip.rs:117` | builds struct field at L133 with `Field::new(format!(...), element_type, true)`, dropping metadata for each zipped column |
| `repeat` | `functions-nested/src/repeat.rs:118` | `Field::new_list_field(element_type, true)` at L121, L125 |
| `struct` | `functions/src/core/struct.rs:102` | constructs `Field::new(format!("c{pos}"), dt.clone(), true)` at L110 with no metadata |

Verified to be **correct** (already override `return_field_from_args` and propagate input field metadata):

- `named_struct` (`functions/src/core/named_struct.rs:98`)
- `coalesce` (input field is returned at L89)
- `array_remove`, `array_remove_n`, `map_values`

Not affected (output is a primitive scalar or input-independent type, so there is no inner field to lose metadata on): `array_sort`, `array_replace`, `array_concat`, `array_append`, `array_prepend`, `array_distinct`, `array_slice`, `array_pop_front`, `array_pop_back`, `set_ops` (union/intersect/except), `min`/`max` on arrays, `string_agg`.

`flatten` (`functions-nested/src/flatten.rs:91`) is borderline: it overrides only `return_type` but the runtime path mostly reuses the original inner field via destructuring, so simple cases preserve metadata. Worth a focused test pass to confirm.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.