elastic / elastic/observability-migration-platform

Datadog: formula-wrapped distributions miss percentile envelope; formula grouping union over-applies subsets

Open
#288 0 comments 0 reactions 0 assignees View on GitHub
asset:dashboards bug program:datadog quality source:datadog workstream:translator
Dominant language
Python
Stars
6
Forks
8
Avg merge
2d 22h
Merged PRs (30d)
23

Description

## Summary

Two related Datadog translator correctness gaps found during a fidelity review of the current branch (`feat/kibana-native-dashboard-ir`). Both live in `observability_migration/adapters/source/datadog/translate.py`.

1. **Formula-wrapped distribution widgets miss the percentile envelope.** Histogram-style / bare distribution timeseries get avg+p50/p90/p99 via `_translate_single_metric`, but widgets that wrap the same metric in `formulas: [{"formula":"query1"}]` go through `_translate_formula_metric_widget` and emit a single aggregate series instead.
2. **Formula grouping union is too broad.** `_ensure_formula_specs_compatible` unions *any* subset mismatch (`{pod}` vs `{pod,host}`), not only empty↔grouped. That incorrectly splits the coarser series and can change chart semantics.

## Gap 1 — formula-wrapped distributions

### Expected

A Datadog `distribution` timeseries widget should migrate to the requested aggregator **plus** p50/p90/p99 percentile series (documented approximation; ES|QL has no native distribution histogram panel).

### Actual

- Path A (`_translate_single_metric`, ~L373–393): `widget.widget_type == "distribution" and is_timeseries` → `_build_distribution_percentile_esql(...)` ✅
- Path B (`_translate_formula_metric_widget`, entered when `requests[].formulas` is present): no distribution-envelope branch → single `STATS` / measure line ❌

### Fixture evidence

`infra/datadog/dashboards/integrations/redis.json` — widget **"Key length distribution"**:

```json
{
"formulas": [{ "formula": "query1" }],
"queries": [{
"query": "sum:redis.key.length{$scope, $host, $key} by {key}",
"data_source": "metrics",
"name": "query1",
"aggregator": "avg",
"semantic_mode": "combined"
}],
"response_format": "scalar"
}
```

Migrated output is a single aggregate series (e.g. `SUM(...)`), not the percentile envelope used for non-formula distributions.

### Suggested fix

Reuse `_build_distribution_percentile_esql` (or equivalent) from the formula path when:

- `widget.widget_type == "distribution"`, and
- the formula is a trivial passthrough (`query1` / single named query with no arithmetic), and
- the panel is timeseries-shaped (same conditions as path A).

Non-trivial formulas (`query1 / query2`, etc.) should keep failing closed or warning rather than inventing an envelope.

## Gap 2 — formula grouping union over-applies

### Where

`_ensure_formula_specs_compatible` (~L1132–1173):

```python
if kibana_type in ("xy", "heatmap") and (
base_set.issubset(other_set) or other_set.issubset(base_set)
):
# unite dimensions on the union
```

### Expected

Union / broadcast is correct for **empty ↔ grouped** (Datadog broadcasts the ungrouped side). Example: `{}` vs `{pod}` → group by `pod`, ungrouped measure repeats per pod.

### Actual

The same branch also accepts **non-empty proper subsets**, e.g. `{pod}` vs `{pod,host}`. Uniting to `{pod,host}` splits the coarser `{pod}` series across hosts, which is not equivalent to Datadog’s resolution of that mismatch.

### Suggested fix

Narrow the automatic union to cases where one side has **no** group fields (empty ↔ non-empty). For non-empty subset mismatches, keep `requires_manual` (or a more precise join strategy if/when proven against Datadog).

## Acceptance criteria

- [ ] Redis (and similar) formula-passthrough distribution widgets emit the same percentile envelope as non-formula distributions, with the existing approximation warning.
- [ ] Unit/snapshot coverage for a formula-wrapped distribution fixture.
- [ ] `_ensure_formula_specs_compatible` only auto-unions empty↔grouped; `{a}` vs `{a,b}` remains manual (or documented equivalent behavior with tests).
- [ ] Existing formula fixtures that rely on empty↔grouped broadcasting still pass.

## Notes

- Grafana side of the same review looked sound for the intended idioms; these two items are Datadog-specific follow-ups.
- Related recent work: distribution percentile envelope in `_translate_single_metric` / `_build_distribution_percentile_esql`; percentile request-aggregator handling; multi-tag XY `series_group`.

Contributor guide

Open the contributing guide

Research direction

Start in observability_migration/adapters/source/datadog/translate.py, comparing _translate_single_metric, _translate_formula_metric_widget, _build_distribution_percentile_esql, and _ensure_formula_specs_compatible. Use infra/datadog/dashboards/integrations/redis.json and existing Datadog unit or snapshot tests as fixtures. Done means passthrough distribution formulas produce the existing percentile envelope and only empty↔grouped specs auto-union, with coverage for both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre, testing, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.