apache / apache/texera

Move dataset file path resolution out of CU Master/Worker

Open
#5,013 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary

Sub-issue of #5011.

Move the `FileResolver.datasetResolveFunc` direct DB call
(`common/workflow-core/.../FileResolver.scala`) behind an HTTP service
that owns the credentials. The executor forwards the originating user's
JWT.

`datasetResolveFunc` joins `USER × DATASET × DATASET_VERSION` to translate
`/owner/dataset/version/file` into a `dataset://///`
URI. It is the only `SqlServer` call site reachable from CU Master /
Worker that is not about execution metadata. It is invoked from
`LogicalPlan.resolveScanSourceOpFileName`, which runs during workflow
compile on every execution.

The natural owner for this lookup is `file-service` (it already owns
the dataset model). Done when no code reachable from CU Master / Worker
calls `SqlServer` for dataset path resolution, the new endpoint is
`@Auth`-checked, and the existing `FileResolverSpec` plus an end-to-end
workflow run that scans a dataset file still pass.

### Task Type

- [x] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other

Contributor guide

Open the contributing guide

Research direction

Start with FileResolver.scala and LogicalPlan.resolveScanSourceOpFileName to trace the dataset path lookup, then inspect file-service's dataset model and existing @Auth-checked endpoints. Use FileResolverSpec and an end-to-end workflow run that scans a dataset file as the validation points. Done means CU Master/Worker no longer reaches SqlServer for this resolution and the lookup forwards the originating user's JWT.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
api, backend, database
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.