elastic / elastic/observability-migration-platform
Datadog: formula-wrapped distributions miss percentile envelope; formula grouping union over-applies subsets
- Dominant language
- Python
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 23
Description
## Summary
Two related Datadog translator correctness gaps found during a fidelity review of the current branch (`feat/kibana-native-dashboard-ir`). Both live in `observability_migration/adapters/source/datadog/translate.py`.
1. **Formula-wrapped distribution widgets miss the percentile envelope.** Histogram-style / bare distribution timeseries get avg+p50/p90/p99 via `_translate_single_metric`, but widgets that wrap the same metric in `formulas: [{"formula":"query1"}]` go through `_translate_formula_metric_widget` and emit a single aggregate series instead.
2. **Formula grouping union is too broad.** `_ensure_formula_specs_compatible` unions *any* subset mismatch (`{pod}` vs `{pod,host}`), not only empty↔grouped. That incorrectly splits the coarser series and can change chart semantics.
## Gap 1 — formula-wrapped distributions
### Expected
A Datadog `distribution` timeseries widget should migrate to the requested aggregator **plus** p50/p90/p99 percentile series (documented approximation; ES|QL has no native distribution histogram panel).
### Actual
- Path A (`_translate_single_metric`, ~L373–393): `widget.widget_type == "distribution" and is_timeseries` → `_build_distribution_percentile_esql(...)` ✅
- Path B (`_translate_formula_metric_widget`, entered when `requests[].formulas` is present): no distribution-envelope branch → single `STATS` / measure line ❌
### Fixture evidence
`infra/datadog/dashboards/integrations/redis.json` — widget **"Key length distribution"**:
```json
{
"formulas": [{ "formula": "query1" }],
"queries": [{
"query": "sum:redis.key.length{$scope, $host, $key} by {key}",
"data_source": "metrics",
"name": "query1",
"aggregator": "avg",
"semantic_mode": "combined"
}],
"response_format": "scalar"
}
```
Migrated output is a single aggregate series (e.g. `SUM(...)`), not the percentile envelope used for non-formula distributions.
### Suggested fix
Reuse `_build_distribution_percentile_esql` (or equivalent) from the formula path when:
- `widget.widget_type == "distribution"`, and
- the formula is a trivial passthrough (`query1` / single named query with no arithmetic), and
- the panel is timeseries-shaped (same conditions as path A).
Non-trivial formulas (`query1 / query2`, etc.) should keep failing closed or warning rather than inventing an envelope.
## Gap 2 — formula grouping union over-applies
### Where
`_ensure_formula_specs_compatible` (~L1132–1173):
```python
if kibana_type in ("xy", "heatmap") and (
base_set.issubset(other_set) or other_set.issubset(base_set)
):
# unite dimensions on the union
```
### Expected
Union / broadcast is correct for **empty ↔ grouped** (Datadog broadcasts the ungrouped side). Example: `{}` vs `{pod}` → group by `pod`, ungrouped measure repeats per pod.
### Actual
The same branch also accepts **non-empty proper subsets**, e.g. `{pod}` vs `{pod,host}`. Uniting to `{pod,host}` splits the coarser `{pod}` series across hosts, which is not equivalent to Datadog’s resolution of that mismatch.
### Suggested fix
Narrow the automatic union to cases where one side has **no** group fields (empty ↔ non-empty). For non-empty subset mismatches, keep `requires_manual` (or a more precise join strategy if/when proven against Datadog).
## Acceptance criteria
- [ ] Redis (and similar) formula-passthrough distribution widgets emit the same percentile envelope as non-formula distributions, with the existing approximation warning.
- [ ] Unit/snapshot coverage for a formula-wrapped distribution fixture.
- [ ] `_ensure_formula_specs_compatible` only auto-unions empty↔grouped; `{a}` vs `{a,b}` remains manual (or documented equivalent behavior with tests).
- [ ] Existing formula fixtures that rely on empty↔grouped broadcasting still pass.
## Notes
- Grafana side of the same review looked sound for the intended idioms; these two items are Datadog-specific follow-ups.
- Related recent work: distribution percentile envelope in `_translate_single_metric` / `_build_distribution_percentile_esql`; percentile request-aggregator handling; multi-tag XY `series_group`.
Contributor guide
Research direction
Start in observability_migration/adapters/source/datadog/translate.py, comparing _translate_single_metric, _translate_formula_metric_widget, _build_distribution_percentile_esql, and _ensure_formula_specs_compatible. Use infra/datadog/dashboards/integrations/redis.json and existing Datadog unit or snapshot tests as fixtures. Done means passthrough distribution formulas produce the existing percentile envelope and only empty↔grouped specs auto-union, with coverage for both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre, testing, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100