apache / apache/texera

Add Prometheus metrics and Grafana dashboards for workflow execution observability

Open
#6,272 5 comments 0 reactions 1 assignee Claimed by @Musxeto View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Texera's Amber engine already collects rich runtime statistics internally (tuple counts, processing times, worker counts, execution duration) via `StatisticsManager` and `ExecutionStatsService` — but these are only visible in the UI during an active session and aren't available for alerting, historical analysis, or system-level correlation.

**This feature adds:**

- A **Prometheus `/metrics` endpoint** on the coordinator service exposing workflow, operator, and JVM metrics
- **Pre-built Grafana dashboards** for workflow overview, operator performance, and system health
- **Docker Compose integration** (opt-in `observability` profile) for one-command setup of Prometheus + Grafana

**Problems this solves:**

- No way to monitor execution health outside the UI
- No alerting for stalled workflows or resource exhaustion
- No historical trend analysis for capacity planning
- JVM/system metrics (heap, GC, threads) are completely invisible

### Proposed Solution or Design

**Instrument the existing stats pipeline** with [Micrometer](https://micrometer.io/) (JVM metrics facade) and expose a standard Prometheus scrape endpoint. No changes to the existing stats interfaces — just an additional export path.

**High-level approach:**

1. Add Micrometer & Prometheus registry dependency to `amber/build.sbt`
2. Create a `MetricsService` singleton wiring existing `StatisticsManager` / `ExecutionStatsService` data into Micrometer meters
3. Expose `GET /api/metrics` via a new Dropwizard resource
4. Add Prometheus + Grafana to `bin/single-node/docker-compose.yml` under an optional `observability` profile
5. Ship 3 pre-provisioned Grafana dashboards (Workflow Overview, Operator Deep-Dive, System Health)

**Key metrics:**
- Workflow: active count, execution duration, state transitions
- Operator: input/output tuple rates, processing time breakdown, worker count
- System: JVM heap, GC pauses, thread count (via Micrometer's built-in JVM binders)

**Contributors:** We (@musxeto and @thegamingbat) would like to work on this. Happy to discuss the design further before coding.

### Affected Area

Workflow Engine (Amber), Deployment / Infrastructure

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.