apache / apache/texera

Operator output port result cache (MVP)

Open
#5,881 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Feature Summary

Texera runs a workflow from the start every time, even when the user changed only one operator near the end. This adds an operator output port result cache: on a re-run, an output port whose upstream computation is unchanged reads its saved result instead of recomputing it, for faster iteration.

The MVP always reuses a matched port's result (full reuse); there is no cost-based reuse decision and no eviction, which are future work. When no port is matched, the system behaves exactly as today, so the work can land inactive and turn on once results are saved.

Design and rationale: Discussion #5880.

### Proposed Solution or Design

This umbrella issue tracks the tasks that deliver the feature (the sub-issues below). They are independent units of work; each changes behavior only when a cache result is matched, so each can be reviewed on the guarantee that an empty cache reproduces current behavior. Real prerequisites are noted in each task.

- #5882 Add the cache storage layer (table and cache key).
- #5883 Carry cache-reuse status in operator metrics.
- [ ] #6729
- #5884 Add cache-reuse planning and skipped-region execution to the scheduler.
- #5885 Wire cache lookup, result saving, REST endpoints, and cleanup into execution.
- #5886 Add the cache panel and canvas display to the workflow editor.

Notes:
- Generated code (jOOQ for the table, protobuf for the metrics flag) is not committed; the changes ship the source and the build regenerates the rest.
- The implementation is re-derived from the prototype branch onto current `main`, not merged from it directly.

### Affected Area

Workflow Engine (Amber), Storage / Metadata, Workflow UI

Contributor guide

Open the contributing guide

Research direction

Start with the independent sub-issues #5882 through #5886 and read Discussion #5880 for the design and rationale. The work is complete when the storage, metrics, scheduler, execution/REST, and workflow-editor tasks are delivered, while an empty cache preserves current behavior and matched results are reused.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, database, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.