apache / apache/texera

Workflow runtime performance visualization (heat-map overlay)

Open
#5,772 6 comments 0 reactions 1 assignee Claimed by @PG1204 View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Texera already collects rich per-operator statistics during (and after) a workflow run: processing time, idle time, input/output row counts and byte sizes, and worker counts. Today these are only readable one operator at a time, so there is no at-a-glance way to see **where the time and data actually went**.

This is an umbrella issue to add a toggleable, canvas-level **heat-map overlay** that colors operators cold -> hot by runtime performance, turning the statistics Texera already produces into an immediate visual answer to "what is my bottleneck?". It is a read-only consumer of existing data and requires **no backend changes**. Follow-up from the RFC discussion in #5216.

- Toggle a "Performance" heat-map layer on the canvas, alongside the existing Grid / Regions / Workers layers
- Color every operator cold -> hot by a normalized performance score
- Switch between three views: Runtime, Throughput, and I/O imbalance
- Read an on-canvas legend showing the cold -> hot scale and the active view
- See colors update live as statistics stream in during a run
- Keep the heat-map after a page refresh, including for an already-finished run, without having to re-execute the workflow

### Proposed Solution or Design

A read-only frontend feature driven by `WorkflowStatusService` as the single source of truth. The execution engine already streams every metric needed over the websocket, and a historical-stats API already exists for restoring a finished run after refresh, so there are **no backend changes**.

### Heat-map Views

| View | Answers | Derived from |
| ------------- | -------------------------------- | ------------------------------------ |
| Runtime | Which operator burns the time? | data + control processing time |
| Throughput | Which operator moves the rows? | output row count |
| I/O imbalance | Which operator over/under-feeds? | output-to-input row ratio |

### Key Behaviors

- Modeled on the existing toggleable canvas layers (Grid / Regions / Workers)
- Single source of truth: all metrics, live and historical, flow through `WorkflowStatusService` - no parallel service
- No visual conflict: the heat-map colors only the operator body fill, while the existing execution-status indicator (operator border) is left untouched
- Normalized scoring with skew handling, so one dominant operator does not flatten the rest of the scale
- Restores the last execution's statistics on load via the existing runtime-statistics API, gated on the overlay being enabled

### Sub-tasks

Delivered as three sequential sub-issues, to be created and linked under this umbrella issue:

1. Capture all per-operator performance metrics in `WorkflowStatusService` (no UI)
2. Add the toggleable heat-map overlay layer with the three views and legend
3. Persist the toggle/view and restore the last run's statistics after refresh

### Out of Scope

Rule-based hints, structural rewrite suggestions, and downloadable reports from the original RFC prototype are intentionally deferred to future discussion.

### Affected Area

Workflow UI

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.