apache / apache/arrow

[C++][Acero] Preserve input nullability in direct field projections

Open
#51,231 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Could Acero preserve input nullability when a `project` only references top-level input fields? Currently, such a projection changes a required input field to nullable in the output schema. A `table_source` and `Table.select` preserve the same field's nullability. The projected values are unchanged.

This also affects Substrait `RelCommon.emit`: even an identity mapping `[0, 1]` changes a required field to nullable, while a bare read preserves it.

### To reproduce

Install `pyarrow==25.0.1` and run:

```python
import pyarrow as pa
import pyarrow.acero as ac
import pyarrow.compute as pc

schema = pa.schema([
pa.field("r", pa.int64(), nullable=False),
pa.field("n", pa.int64(), nullable=True),
])
table = pa.Table.from_arrays([
pa.array([1, 2], type=pa.int64()),
pa.array([None, 3], type=pa.int64()),
], schema=schema)
source = ac.Declaration("table_source", ac.TableSourceNodeOptions(table))
projected = ac.Declaration(
"project",
ac.ProjectNodeOptions([pc.field("r"), pc.field("n")], ["r", "n"]),
inputs=[source],
).to_table(use_threads=False)

for name, value in [
("source", source.to_table(use_threads=False)),
("select", table.select([0, 1])),
("project", projected),
]:
print(name, [field.nullable for field in value.schema])
print("values unchanged:", projected.to_pylist() == table.to_pylist())
```

Output:

```text
source [False, True]
select [False, True]
project [True, True]
values unchanged: True
```

I expected the direct-reference projection to retain `[False, True]`. It only selects the two input fields; there are no functions or casts, and the required input field contains no nulls.

### Schema compatibility impact

Writing the projected table to an IPC stream created with the original schema fails. Run this after the example above:

```python
import pyarrow.ipc as ipc

with ipc.new_stream(pa.BufferOutputStream(), schema) as writer:
writer.write_table(projected)
```

This raises `ArrowInvalid: Tried to write record batch with different schema`. Replacing `projected` with `table` or `table.select([0, 1])` succeeds. The values and column order are identical; the schema differs only in nullability.

### Where it happens

In [ProjectNode::Make](https://github.com/apache/arrow/blob/apache-arrow-25.0.1/cpp/src/arrow/acero/project_node.cc#L70), output fields are created with the expression name and type, without carrying input nullability. The same construction is present at main commit `397b3d0023030f6c6bc69d214ea7b27a687256f8`; the runtime reproduction above is against 25.0.1.

Substrait's [ProcessEmit](https://github.com/apache/arrow/blob/apache-arrow-25.0.1/cpp/src/arrow/engine/substrait/relation_internal.cc#L136) adds a field-reference project, which explains the identity emit result. Related issue #20108 concerns Substrait type serialization and deserialization; the native Acero example above reproduces this behavior without either step.

The same direct-projection behavior was already [reported in #35730](https://github.com/apache/arrow/issues/35730#issuecomment-1562446055). That issue was closed by #35860, which added a custom output schema for dataset writes. It did not change `ProjectNode`. This request follows up on preserving input nullability in direct projections.

The [Acero Substrait documentation](https://github.com/apache/arrow/blob/apache-arrow-25.0.1/docs/source/cpp/acero/substrait.rst#types) notes incomplete support for non-nullable types. I have not found an explicit guarantee that `ProjectNode` preserves input nullability, so I am requesting a schema-preservation improvement here. This request is limited to direct references to top-level input fields; general expression nullability inference is outside its scope.

Four related Substrait cases, including identity and reordered emit with controls, are [in the corpus](https://github.com/alexandrefimov/substrait-conformance-cases/tree/6f5af2ed59502c38189a4e4afeff79732df53626/probe/structural-cases/acero).

Contributor guide

Open the contributing guide

Research direction

Start in cpp/src/arrow/acero/project_node.cc at ProjectNode::Make and inspect the output-field construction; compare field-reference handling in cpp/src/arrow/engine/substrait/relation_internal.cc at ProcessEmit. Reproduce the supplied Python example and verify direct and identity projections retain [False, True] and remain writable with the original IPC schema, without expanding into general expression nullability inference.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.