Move execution-related metadata operations out of CU Master/Worker
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 209
Description
### Task Summary
Sub-issue of #5011.
Move every direct `SqlServer` call reachable from CU Master / CU Worker
that touches the **execution metadata tables** (i.e. `workflow_executions`
and the operator/port URI registry) behind an HTTP service that owns
the credentials. The executor forwards the originating user's JWT.
Code paths in scope (per the Current Usage table in #5011):
- Execution lifecycle — `web/service/WorkflowService.scala` (INSERT new
row, UPDATE on every status / log_location / runtime_stats_uri / result change).
- State transitions — `web/storage/ExecutionStateStore.scala`
(`updateWorkflowState` from many sites in `WorkflowService` /
`WorkflowExecutionService`).
- Operator/port URI registry — `web/resource/.../WorkflowExecutionsResource.scala`
(`insertOperatorPortResultUri`, `insertOperatorConsoleUri`,
`getResultUriByLogicalPortId`, `getLatestExecutionID`, …). Called by
engine code (e.g. `RegionExecutionCoordinator`) and by
`SyncExecutionResource`.
- Result export — `web/resource/.../WorkflowExecutionsResource.scala`
(`exportResultToDataset`, `exportResultToLocal`) →
`web/service/ResultExportService.scala`. Endpoints today are hosted on
CU Master; their DB lookups go through the same registry calls above.
- Result/log cleanup — `web/ComputingUnitMaster.scala` (`cleanExecutions`,
`recurringCheckExpiredResults`).
- Cost-based scheduling — `engine/architecture/scheduling/CostEstimator.scala`
(`getOperatorExecutionTimeInSeconds` reads the latest successful
`workflow_executions.runtime_stats_uri` for a `wid`).
Done when no engine or service code reachable from CU Master / Worker
calls `SqlServer` for execution-metadata reads or writes, every former
DB call has an HTTP endpoint on web-app with `@Auth`-checked JWT
authorization, and the existing test suite plus an end-to-end run still
pass.
### Task Type
- [x] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other
Contributor guide
Research direction
Read #5011's Current Usage table, then trace the listed paths: WorkflowService.scala, ExecutionStateStore.scala, WorkflowExecutionsResource.scala, ResultExportService.scala, ComputingUnitMaster.scala, and CostEstimator.scala. Run the existing test suite before changing anything. Done means execution-metadata SQL calls are replaced by authorized web-app endpoints, with the existing suite and an end-to-end run passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- api, backend, database
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100