apache / apache/texera

Move execution-related metadata operations out of CU Master/Worker

Open
#5,012 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
2d 8h
Merged PRs (30d)
209

Description

### Task Summary

Sub-issue of #5011.

Move every direct `SqlServer` call reachable from CU Master / CU Worker
that touches the **execution metadata tables** (i.e. `workflow_executions`
and the operator/port URI registry) behind an HTTP service that owns
the credentials. The executor forwards the originating user's JWT.

Code paths in scope (per the Current Usage table in #5011):

- Execution lifecycle — `web/service/WorkflowService.scala` (INSERT new
row, UPDATE on every status / log_location / runtime_stats_uri / result change).
- State transitions — `web/storage/ExecutionStateStore.scala`
(`updateWorkflowState` from many sites in `WorkflowService` /
`WorkflowExecutionService`).
- Operator/port URI registry — `web/resource/.../WorkflowExecutionsResource.scala`
(`insertOperatorPortResultUri`, `insertOperatorConsoleUri`,
`getResultUriByLogicalPortId`, `getLatestExecutionID`, …). Called by
engine code (e.g. `RegionExecutionCoordinator`) and by
`SyncExecutionResource`.
- Result export — `web/resource/.../WorkflowExecutionsResource.scala`
(`exportResultToDataset`, `exportResultToLocal`) →
`web/service/ResultExportService.scala`. Endpoints today are hosted on
CU Master; their DB lookups go through the same registry calls above.
- Result/log cleanup — `web/ComputingUnitMaster.scala` (`cleanExecutions`,
`recurringCheckExpiredResults`).
- Cost-based scheduling — `engine/architecture/scheduling/CostEstimator.scala`
(`getOperatorExecutionTimeInSeconds` reads the latest successful
`workflow_executions.runtime_stats_uri` for a `wid`).

Done when no engine or service code reachable from CU Master / Worker
calls `SqlServer` for execution-metadata reads or writes, every former
DB call has an HTTP endpoint on web-app with `@Auth`-checked JWT
authorization, and the existing test suite plus an end-to-end run still
pass.

### Task Type

- [x] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Research direction

Read #5011's Current Usage table, then trace the listed paths: WorkflowService.scala, ExecutionStateStore.scala, WorkflowExecutionsResource.scala, ResultExportService.scala, ComputingUnitMaster.scala, and CostEstimator.scala. Run the existing test suite before changing anything. Done means execution-metadata SQL calls are replaced by authorized web-app endpoints, with the existing suite and an end-to-end run passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
api, backend, database
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.