[Bug] Support schema-evolved multi-group Measure aggregation
- Dominant language
- Java
- Stars
- 25k
- Forks
- 6.6k
- Avg merge
- 10h 5m
- Merged PRs (30d)
- 16
Description
## Summary
Make valid aggregate queries succeed when participating storage groups have different projected schemas. Follow-up to #14066 under #13998.
## Size audit and dependencies
Classification: **tracking parent**, not an executable leaf or automated implementation task.
Boundary: multi-group `DistributedPlan.Execute` through `executeAgg`, `ReduceRawFrames`, and `ReducePartialBatches`.
Schema alignment and storage-group-aware replica handling are independently testable changes; split them into executable tasks before implementation.
Blocked by: #14066 via BanyanDB PR apache/skywalking-banyandb#1326, to finalize the remaining boundary after legacy removal.
## Proposed initial slice
Normalize missing **projected, non-grouping tags** to null before partial reduction, rather than rejecting the query because partial layouts differ. Do not simply disable structural schema validation.
## Concrete oracle
- Old schema: `svc,value`; rows `(old,2)` and `(old,3)`.
- New schema: `svc,zone,value`; rows `(new,west,7)` and `(new,west,11)`.
- Query: `GROUP BY svc`, `SUM(value)`, project `svc,zone`.
Expected unordered output:
```text
old, NULL, 5
new, west, 18
```
Different service keys deliberately avoid ambiguity over representative values of non-grouping tags. Currently differently shaped partial batches can fail with `partial ... schema mismatch`.
## Completion criteria
- [ ] The schema-evolution oracle succeeds through the distributed production caller, not only a reducer helper; add a regression that fails on the current structural mismatch.
- [ ] Normalization preserves aggregate-state columns, including count sidecars used by MEAN, and rejects genuinely incompatible types with a clear error.
- [ ] Replica deduplication preserves storage-group provenance: contributions 5 and 18 for service `api` from two different storage groups must total **23**, even when both groups use shard 0; duplicate replicas of either contribution must not increase that sum. This requires its own focused implementation/test slice.
- [ ] Each executable child activates the real distributed caller and has independently specified expected results.
Relevant code: `pkg/query/vectorized/measure/reduce.go`, `pkg/query/vectorized/measure/plan/distributed.go`; extend distributed-plan and topology-matrix coverage.
```sh
go test ./pkg/query/vectorized/measure/...
go test ./test/integration/distributed/query/...
```
## Out of scope of the first slice
Missing **GroupBy keys**: these fail earlier during planning and need a separately specified change, not an assumed reducer-only fix. Also exclude changing aggregate/operator semantics or replacing the ordinary multi-group row merger.
Parent: #13998.
## Compatibility and review policy
- Preserve existing on-disk and wire formats and mixed-version compatibility. Any necessary protocol change requires a separately reviewed compatibility contract.
- Use explicit, hand-calculated expected results; do not depend on the removed row engine as the test oracle.
- Human review enforces the no-new-row-path policy. A row-call-site lint guard is not required.
Contributor guide
Research direction
Treat this as a tracking parent rather than an executable task. Start by reading pkg/query/vectorized/measure/reduce.go and pkg/query/vectorized/measure/plan/distributed.go, then run the listed Go test commands and review existing distributed-plan and topology-matrix coverage. Done requires splitting the schema-alignment and replica-handling work into independently specified child tasks with production-caller regressions and explicit expected results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100