Wire cache lookup, result saving, REST endpoints, and cleanup into execution
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Task Summary
Part of #5881 (design: #5880).
Connect the pieces so the cache is used: look up matched ports at submission, save a result when an output port finishes, expose the cache endpoints, and remove entries when executions are deleted. After this task the feature works on the backend.
**Scope**
- Submission-time matching: `WorkflowExecutionService` asks the cache service for the matched ports and stores them on `WorkflowSettings.cachedOutputs`, which the scheduler task (#5884) reads.
- Saving on completion: when an output port finishes, `PortCompletedHandler` sends a new `PortMaterialized` event; `ExecutionCacheService` receives it and writes a cache entry. The engine sends an event and the service layer handles it, matching the existing pattern for statistics and worker updates, so the engine stays unaware of the web layer.
- Two small websocket events so the UI can show cache entries and which ones the current run can use, plus the state stores that hold them; both default to empty.
- Cache endpoints on `WorkflowExecutionsResource`: list entries, clear all, clear for a selected operator, and remove entries whose cache key no longer matches the current workflow. The result location is left out of the listing.
- Cleanup: when executions are deleted, or a computing unit is torn down, the cache entries those executions produced are removed.
**Behavior change to call out.** To make reuse possible, this stops eagerly clearing the previous run's results when a new run starts. This affects re-runs whether or not the cache is used, so it must be described clearly.
**Why the rest is safe with no matched ports.** The lookup returns nothing when the table is empty, so `cachedOutputs` stays empty and the scheduler behaves as before. The saving path is wrapped so a write failure logs and does not fail the run. The endpoints are new routes, and the cleanup is a no-op with no rows.
**Depends on.** #5882 (cache table and service) and #5884 (the scheduler that reads `cachedOutputs`). Activates the feature.
**Out of scope.** The cache panel and canvas display (#5886). Any cost-based or eviction logic.
### Task Type
- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [x] Other
Contributor guide
Research direction
Start by reading dependencies #5882 and #5884, then trace the named entry points: WorkflowExecutionService, PortCompletedHandler, ExecutionCacheService, WorkflowExecutionsResource, and WorkflowSettings.cachedOutputs. Done means submission lookup, completion saving, websocket state, cache routes, and deletion cleanup are connected, while failed cache writes do not fail executions and reruns retain prior results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- api, backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100