Datadog Hard to Migrate to Elastic use cases

This meta issue was created to contain the full output from AI and the list of related use cases.

This is the complete output from GenAI that will get a sub-issue for each item on this list.

### AI prompt
Unknown what was provided to Google Gemini

### AI response
Migrating from Datadog to Elastic (ELK) can be a headache, primarily because Datadog’s "Monitor" logic often relies on pre-aggregated metrics and specialized mathematical functions that don't always have a 1:1 mapping in Elasticsearch's DSL or Kibana's Alerting rules.
Here are five complex Datadog alert queries in JSON format, focusing on features like **Anomaly Detection, Sparse Data Fill, and Cross-Metric Math.**

#### 1. The Seasonality-Aware Anomaly Detection
This query uses Datadog’s proprietary anomalies() function, which applies a seasonal trend decomposition. Recreating this in Elastic requires configuring a specific Machine Learning (ML) job, as standard Elastic alerts don't handle triple-exponential smoothing natively within a simple query.

```json
{
"name": "Seasonal Anomaly: Checkout Latency",
"type": "query alert",
"query": "avg(last_4h):anomalies(avg:ecommerce.checkout.latency{env:prod}, 'agile', 2, direction='both', alert_window='last_15m', interval=60, count_default_zero='true') >= 1",
"message": "Checkout latency is deviating from historical weekly patterns. @ops-team",
"options": {
"thresholds": { "critical": 1 }
}
}
```

#### 2. The "Sparse Data" Fill Logic
In Datadog, .fill(zero) or .fill(last) is handled at the query level. Elastic's bucket_script or gap_policy behaves differently. If you are comparing two sparse metrics using division, the logic often breaks during migration.

```json
{
"name": "Error Rate with Sparse Data Handling",
"type": "query alert",
"query": "avg(last_5m):default_zero(sum:api.errors{*} / sum:api.requests{*}) > 0.05",
"message": "High error rate detected. Note: default_zero used to prevent null-pointer logic. {{value}}",
"options": {
"thresholds": { "critical": 0.05 }
}
}
```

#### 3. Cross-Metric Ratio with Dimensional Filtering
Datadog allows you to perform math across two entirely different metric names and then filter by a shared tag. In Elastic, this usually requires a complex filter_ratio in TSVB or an ES|QL query which is structured very differently from this one-liner.

```json
{
"name": "Memory Usage vs Limit Ratio",
"type": "query alert",
"query": "avg(last_5m):( sum:container.memory.usage{cluster:prod} / sum:container.memory.limit{cluster:prod} ) * 100 > 90",
"message": "Cluster {{cluster.name}} is approaching memory limits. Current: {{value}}%",
"options": {
"thresholds": { "critical": 90 }
}
}
```

#### 4. Forecast-Based Warning (Predictive Alerting)
The forecast() function predicts where a metric will be in the future based on past behavior. Elastic requires the "Forecasting" feature within their ML module, which isn't a simple JSON query string—it's a separate stateful job.

```json
{
"name": "Predictive Disk Space Exhaustion",
"type": "query alert",
"query": "avg(last_1h):forecast(avg:system.disk.used{host:db-master}, 'linear', 3, interval='60s', history='7d') >= 0.95",
"message": "Disk is predicted to hit 95% capacity within the next 3 hours. @dba",
"options": {
"thresholds": { "critical": 1 }
}
}
```

#### 5. Change Alert (Relative Delta)
This monitors the difference between the current value and the value from an hour ago. While Elastic can do "Serial Differencing," Datadog’s change() function is much more intuitive for end-users, making the translation of the "relative" versus "absolute" logic tricky.

```json
{
"name": "Sudden Drop in Active Users",
"type": "query alert",
"query": "change(avg(last_5m):sum:app.users.active{*} , 'last_1h') < -50",
"message": "Active user count dropped by more than 50 compared to an hour ago.",
"options": {
"thresholds": { "critical": -50 }
}
}
```

#### Why these are difficult to migrate:

- **Proprietary Algorithms:** Functions like agile or robust in Datadog's anomaly detection are black boxes. Elastic’s ML uses different statistical models.
- **Evaluation Engine:** Datadog evaluates queries on its own ingest servers. Elastic alerts (Kibana) have to query the Elasticsearch indices, which can lead to timing and "heavy query" performance issues for complex math.
- **JSON Structure:** Datadog uses a flat string query (avg(last_5m):...), whereas Elastic alerts often require multi-layered JSON DSL or specific "Aggregations."