[C++][Acero] Preserve input nullability in direct field projections
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Could Acero preserve input nullability when a `project` only references top-level input fields? Currently, such a projection changes a required input field to nullable in the output schema. A `table_source` and `Table.select` preserve the same field's nullability. The projected values are unchanged.
This also affects Substrait `RelCommon.emit`: even an identity mapping `[0, 1]` changes a required field to nullable, while a bare read preserves it.
### To reproduce
Install `pyarrow==25.0.1` and run:
```python
import pyarrow as pa
import pyarrow.acero as ac
import pyarrow.compute as pc
schema = pa.schema([
pa.field("r", pa.int64(), nullable=False),
pa.field("n", pa.int64(), nullable=True),
])
table = pa.Table.from_arrays([
pa.array([1, 2], type=pa.int64()),
pa.array([None, 3], type=pa.int64()),
], schema=schema)
source = ac.Declaration("table_source", ac.TableSourceNodeOptions(table))
projected = ac.Declaration(
"project",
ac.ProjectNodeOptions([pc.field("r"), pc.field("n")], ["r", "n"]),
inputs=[source],
).to_table(use_threads=False)
for name, value in [
("source", source.to_table(use_threads=False)),
("select", table.select([0, 1])),
("project", projected),
]:
print(name, [field.nullable for field in value.schema])
print("values unchanged:", projected.to_pylist() == table.to_pylist())
```
Output:
```text
source [False, True]
select [False, True]
project [True, True]
values unchanged: True
```
I expected the direct-reference projection to retain `[False, True]`. It only selects the two input fields; there are no functions or casts, and the required input field contains no nulls.
### Schema compatibility impact
Writing the projected table to an IPC stream created with the original schema fails. Run this after the example above:
```python
import pyarrow.ipc as ipc
with ipc.new_stream(pa.BufferOutputStream(), schema) as writer:
writer.write_table(projected)
```
This raises `ArrowInvalid: Tried to write record batch with different schema`. Replacing `projected` with `table` or `table.select([0, 1])` succeeds. The values and column order are identical; the schema differs only in nullability.
### Where it happens
In [ProjectNode::Make](https://github.com/apache/arrow/blob/apache-arrow-25.0.1/cpp/src/arrow/acero/project_node.cc#L70), output fields are created with the expression name and type, without carrying input nullability. The same construction is present at main commit `397b3d0023030f6c6bc69d214ea7b27a687256f8`; the runtime reproduction above is against 25.0.1.
Substrait's [ProcessEmit](https://github.com/apache/arrow/blob/apache-arrow-25.0.1/cpp/src/arrow/engine/substrait/relation_internal.cc#L136) adds a field-reference project, which explains the identity emit result. Related issue #20108 concerns Substrait type serialization and deserialization; the native Acero example above reproduces this behavior without either step.
The same direct-projection behavior was already [reported in #35730](https://github.com/apache/arrow/issues/35730#issuecomment-1562446055). That issue was closed by #35860, which added a custom output schema for dataset writes. It did not change `ProjectNode`. This request follows up on preserving input nullability in direct projections.
The [Acero Substrait documentation](https://github.com/apache/arrow/blob/apache-arrow-25.0.1/docs/source/cpp/acero/substrait.rst#types) notes incomplete support for non-nullable types. I have not found an explicit guarantee that `ProjectNode` preserves input nullability, so I am requesting a schema-preservation improvement here. This request is limited to direct references to top-level input fields; general expression nullability inference is outside its scope.
Four related Substrait cases, including identity and reordered emit with controls, are [in the corpus](https://github.com/alexandrefimov/substrait-conformance-cases/tree/6f5af2ed59502c38189a4e4afeff79732df53626/probe/structural-cases/acero).
Contributor guide
Research direction
Start in cpp/src/arrow/acero/project_node.cc at ProjectNode::Make and inspect the output-field construction; compare field-reference handling in cpp/src/arrow/engine/substrait/relation_internal.cc at ProcessEmit. Reproduce the supplied Python example and verify direct and identity projections retain [False, True] and remain writable with the original IPC schema, without expanding into general expression nullability inference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100