Move dataset file path resolution out of CU Master/Worker
- Dominant language
- Scala
- Stars
- 314
- Forks
- 187
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 214
Description
### Task Summary
Sub-issue of #5011.
Move the `FileResolver.datasetResolveFunc` direct DB call
(`common/workflow-core/.../FileResolver.scala`) behind an HTTP service
that owns the credentials. The executor forwards the originating user's
JWT.
`datasetResolveFunc` joins `USER × DATASET × DATASET_VERSION` to translate
`/owner/dataset/version/file` into a `dataset://///`
URI. It is the only `SqlServer` call site reachable from CU Master /
Worker that is not about execution metadata. It is invoked from
`LogicalPlan.resolveScanSourceOpFileName`, which runs during workflow
compile on every execution.
The natural owner for this lookup is `file-service` (it already owns
the dataset model). Done when no code reachable from CU Master / Worker
calls `SqlServer` for dataset path resolution, the new endpoint is
`@Auth`-checked, and the existing `FileResolverSpec` plus an end-to-end
workflow run that scans a dataset file still pass.
### Task Type
- [x] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other
Contributor guide
Research direction
Start with FileResolver.scala and LogicalPlan.resolveScanSourceOpFileName to trace the dataset path lookup, then inspect file-service's dataset model and existing @Auth-checked endpoints. Use FileResolverSpec and an end-to-end workflow run that scans a dataset file as the validation points. Done means CU Master/Worker no longer reaches SqlServer for this resolution and the lookup forwards the originating user's JWT.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- api, backend, database
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100