facebookincubator / facebookincubator/velox
Spark aggregation fuzzer crashes streaming aggregation
- Dominant language
- C++
- Stars
- 4.2k
- Forks
- 1.6k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 246
Description
## Summary
The Spark aggregation fuzzer consistently crashes (SIGSEGV at address 0x0) when testing the `first` aggregate function with complex ROW-typed group-by keys in a specific plan variant: **FINAL STREAMING aggregation over LocalMerge with forced abandon-partial-aggregation** (plan #14 in the fuzzer).
## Reproduction
The crash has been confirmed across multiple CI runs with different seeds:
| CI Run | Initial Seed | Crash Iteration | Crash Seed | Input Type | Group-by Keys |
|--------|-------------|-----------------|------------|------------|---------------|
| [22654073141](https://github.com/facebookincubator/velox/actions/runs/22654073141) | 5500 | 309 | 3776397994 | BIGINT | ROW, ROW, ROW, BIGINT |
| [22636891751](https://github.com/facebookincubator/velox/actions/runs/22636891751) | 9886 | 170 | 1997082133 | REAL | ROW(7 fields), ROW(6 fields), ARRAY |
## Failing Plan Shape
The crash always occurs on the same plan structure (plan #14 in the fuzzer):
```
Aggregation[FINAL STREAMING [g0, g1, ...] a0 := first("a0")]
LocalMerge[g0 ASC, g1 ASC, ...]
Aggregation[PARTIAL STREAMING [...] a0 := first(ROW["c0"])]
OrderBy[g0 ASC, g1 ASC, ...]
Project[...]
Values[300 rows in 3 vectors]
Aggregation[PARTIAL STREAMING [...] a0 := first(ROW["c0"])]
OrderBy[...]
Project[...]
Values[300 rows in 3 vectors]
... (4 parallel sources)
```
The crash specifically occurs during the **"forced abandon-partial-aggregation"** test variant of this plan.
## Crash Details
- Signal: SIGSEGV (Signal 11) at address 0x0 (NULL pointer dereference)
- The crash is in the main thread
- Exit code: 139
- No stack trace available in CI binary (lacks debug symbols)
## Analysis
- The `first` aggregate function itself appears to work correctly for simpler plan variants (single aggregation, partial→final, partial→intermediate→final, and even streaming with spilling).
- The crash is specific to the combination of: streaming aggregation + LocalMerge + abandon-partial-aggregation.
- The bug is likely in the aggregation framework's handling of the abandon-partial path for streaming aggregation with LocalMerge, not in the `first` function implementation itself.
- Complex ROW-typed group-by keys may be a contributing factor (possibly related to comparison or serialization of complex types during the merge).
## Workaround
Adding `first` to the `skipFunctions` set in `SparkAggregationFuzzerTest.cpp` will prevent the crash until the root cause is fixed.
Contributor guide
Assessment
This issue has not been assessed yet.