apache / apache/texera

add unit test coverage for DistributedAggregation

Open
#5,777 1 comment 0 reactions 1 assignee Claimed by @lalalastella View on GitHub
starter-task
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

Add a dedicated `DistributedAggregationSpec.scala` that pins the four-function distributed-aggregation contract end-to-end using a representative aggregation (e.g. average): local partials computed per "node" and then merged must equal a single-node fold.

## Background

`DistributedAggregation[P <: AnyRef]` (`operator/aggregate/DistributedAggregation.scala`) is the case class that defines how an aggregate is computed in a data-parallel engine, via four functions (pattern from the SOSP'09 *Distributed Aggregation* paper). It has no dedicated unit-spec.

```scala
case class DistributedAggregation[P <: AnyRef](
init: () => P, // initial partial
iterate: (P, Tuple) => P, // accumulate one input tuple
merge: (P, P) => P, // combine two partials
finalAgg: (P) => Object // partial -> final value
)
```

## Behavior to pin

Define a representative average aggregation `DistributedAggregation[(Double, Long)]` over a single numeric column and assert:

| Step | Contract |
| --- | --- |
| `init()` | returns the identity partial `(0.0, 0L)` |
| `iterate` | folds a tuple's value in: `(sum + v, count + 1)` |
| `merge` | combines two partials additively; commutative/associative |
| `finalAgg` | `(sum, count) => sum / count` |
| distributed == single-node | split the input tuples across two partitions, `iterate` each locally from `init()`, `merge` the partials, `finalAgg` → equals the average from folding all tuples in one partition |
| empty partition | a partition with no tuples contributes `init()` and leaves the merged result unchanged |

Build `Tuple`s with the `Schema` / `Attribute` / `Tuple` helpers — see `AggregateOpSpec` in the same package for the pattern.

## Scope

- New spec: `DistributedAggregationSpec.scala` under `common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/aggregate/`.
- The spec supplies its own representative aggregation functions — the goal is to pin the case class's contract/wiring, not any specific operator.
- No production-code changes.

### Task Type

- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [x] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.