apache / apache/arrow

[Python][C++][Parquet] null_count statistics undercount for a fixed-width leaf under list<struct<...>> when lists are null or empty

Open
#51,097 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Component: Python
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When a leaf column sits under `struct` under a repeated ancestor (`list`, `large_list`, `map`) and some of the containers are null or empty, the Parquet writer undercounts nulls in the column-chunk statistics for **fixed-width** leaves. The `BYTE_ARRAY` leaf next to it in the same struct is correct, so two leaves of the same struct disagree about how many of the same 5 level records are null.

The values themselves round-trip correctly — this is the statistics only.

Reproducer (pyarrow 25.0.1, macOS arm64, also present in the C++ source on `main`):

```python
import pyarrow as pa
import pyarrow.parquet as pq

typ = pa.list_(pa.struct([("s", pa.string()), ("i32", pa.int32())]))
data = [
[{"s": "a", "i32": 1}], # one element
None, # null list
[], # empty list
[{"s": None, "i32": None}, {"s": "b", "i32": 2}], # two elements, one all-null
]
tab = pa.table({"col": pa.array(data, type=typ)})
pq.write_table(tab, "x.parquet")

rg = pq.ParquetFile("x.parquet").metadata.row_group(0)
for i in range(rg.num_columns):
c = rg.column(i)
print(c.path_in_schema, c.physical_type, "levels =", c.num_values,
"null_count =", c.statistics.null_count)
print("roundtrip equal:", pq.read_table("x.parquet").equals(tab))
```

Output:

```
col.list.element.s BYTE_ARRAY levels = 5 null_count = 3
col.list.element.i32 INT32 levels = 5 null_count = 1
roundtrip equal: True
```

Both leaves have the same 5 level records: one value, a null list, an empty list, and two more values of which one is null. Three of the five are null for both leaves. `s` reports 3, `i32` reports 1 — the null-list and empty-list slots are counted as present.

`pyarrow`'s `statistics.num_values` is derived as `chunk num_values - null_count`, so it is wrong in the same way (4 instead of 2). DuckDB 1.5.5's `parquet_metadata()` reports the same `stats_null_count` values (3 and 1), which confirms the numbers are what is written to the file, not an artifact of the pyarrow statistics reader.

The trigger is specifically a fixed-width leaf **under a struct under a repeated ancestor**. Tested with the same four rows:

| shape | result |
|---|---|
| `list>` | `s` correct, `i32` wrong |
| `large_list>` | same |
| `map>` | key correct, value `s` correct, value `i32` wrong |
| `list` (no struct) | correct |
| `list>` | correct |
| `struct>` (no repeated ancestor) | correct |
| top-level `struct` | correct |

Every fixed-width physical type under `list>` is affected: `INT32`, `INT64`, `DOUBLE`, `BOOLEAN`, `FIXED_LEN_BYTE_ARRAY`. It also does not need both cases — a null list alone, or an empty list alone, is enough. It reproduces with `data_page_version` `1.0` and `2.0`.

### Where it comes from

`MaybeCalculateValidityBits` in `cpp/src/parquet/column_writer.cc` has two branches. When `bits_buffer_ == nullptr` it computes

```cpp
*null_count = batch_size - *out_values_to_write;
```

over all `batch_size` levels, which is right. When `bits_buffer_ != nullptr` — which per its own comment is exactly the case where "at least one level of nullable structs directly precede the leaf node" — it instead takes `null_count` from `DefLevelsToBitmap`:

```cpp
internal::DefLevelsToBitmap(def_levels, batch_size, level_info_, &io);
*out_values_to_write = io.values_read - io.null_count;
*out_spaced_values_to_write = io.values_read;
*null_count = io.null_count;
```

`io.null_count` counts nulls only among the *spaced* values, i.e. levels with `def_level >= repeated_ancestor_def_level`. It excludes the empty- and null-container slots.

That value is then handed to `WriteValuesSpaced` from `WriteBatchSpacedInternal`, and `WriteValuesSpaced`'s own docstring states the contract it is meant to satisfy:

```
/// \param num_nulls number of nulls in the values buffer as well as nulls from the
/// ancestor (e.g. empty lists).
```

The `BYTE_ARRAY` specialisation does not go through that path. `TypedColumnWriterImpl::WriteArrowDense` recomputes the count itself against the level count:

```cpp
// Null values in ancestors count as nulls.
const int64_t non_null = data_slice->length() - data_slice->null_count();
...
page_statistics_->IncrementNullCount(batch_size - non_null);
```

which is why the string leaf is right and the fixed-width leaf beside it is not.

This looks like the residual of PARQUET-2067 / #42980 (fixed by #11281, "Fix Parquet null count stats for enclosing null lists"). That fix covered the `bits_buffer_ == nullptr` branch and the `BYTE_ARRAY` path — `list` is correct today — but the `bits_buffer_ != nullptr` branch, reached when a nullable struct sits between the repeated ancestor and the leaf, still uses the narrower count.

The same `null_count` is also passed to `CommitWriteAndCheckPageLimit`, so the page-level counts are likely affected too; I did not verify that directly, as pyarrow does not expose the `DataPageV2` header or `ColumnIndex` null counts.

### Impact

Engines that prune on `null_count` — e.g. deciding a row group cannot satisfy `IS NULL`, or that a page has no nulls — can make the wrong call on such a column. Anything that validates a file's statistics against its data will also see the two leaves of one struct contradict each other.

### Component(s)

C++, Parquet, Python

Contributor guide

Open the contributing guide

Research direction

Start in cpp/src/parquet/column_writer.cc at MaybeCalculateValidityBits, then trace the null_count passed through WriteBatchSpacedInternal and WriteValuesSpaced. Reproduce the issue with the Python list example and add regression coverage for fixed-width leaves under nullable or empty repeated containers. Done means the fixed-width and BYTE_ARRAY leaves report matching correct null counts while values still round-trip.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.