apache / apache/texera

Wire cache lookup, result saving, REST endpoints, and cleanup into execution

Open
#5,885 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

Part of #5881 (design: #5880).

Connect the pieces so the cache is used: look up matched ports at submission, save a result when an output port finishes, expose the cache endpoints, and remove entries when executions are deleted. After this task the feature works on the backend.

**Scope**
- Submission-time matching: `WorkflowExecutionService` asks the cache service for the matched ports and stores them on `WorkflowSettings.cachedOutputs`, which the scheduler task (#5884) reads.
- Saving on completion: when an output port finishes, `PortCompletedHandler` sends a new `PortMaterialized` event; `ExecutionCacheService` receives it and writes a cache entry. The engine sends an event and the service layer handles it, matching the existing pattern for statistics and worker updates, so the engine stays unaware of the web layer.
- Two small websocket events so the UI can show cache entries and which ones the current run can use, plus the state stores that hold them; both default to empty.
- Cache endpoints on `WorkflowExecutionsResource`: list entries, clear all, clear for a selected operator, and remove entries whose cache key no longer matches the current workflow. The result location is left out of the listing.
- Cleanup: when executions are deleted, or a computing unit is torn down, the cache entries those executions produced are removed.

**Behavior change to call out.** To make reuse possible, this stops eagerly clearing the previous run's results when a new run starts. This affects re-runs whether or not the cache is used, so it must be described clearly.

**Why the rest is safe with no matched ports.** The lookup returns nothing when the table is empty, so `cachedOutputs` stays empty and the scheduler behaves as before. The saving path is wrapped so a write failure logs and does not fail the run. The endpoints are new routes, and the cleanup is a no-op with no rows.

**Depends on.** #5882 (cache table and service) and #5884 (the scheduler that reads `cachedOutputs`). Activates the feature.

**Out of scope.** The cache panel and canvas display (#5886). Any cost-based or eviction logic.

### Task Type

- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [x] Other

Contributor guide

Open the contributing guide

Research direction

Start by reading dependencies #5882 and #5884, then trace the named entry points: WorkflowExecutionService, PortCompletedHandler, ExecutionCacheService, WorkflowExecutionsResource, and WorkflowSettings.cachedOutputs. Done means submission lookup, completion saving, websocket state, cache routes, and deletion cleanup are connected, while failed cache writes do not fail executions and reruns retain prior results.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
api, backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.