microsoft / microsoft/dbt-scope
feat: ML-heuristic based AU assignment that optimizes for processing time
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3
- Forks
- 4
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 1
Description
- We know per table, per
.SSfile, num rows - We have the user guess the AU to assign, is dumb
- We could have a
is_au_telemetry_on, which stores AU vs files vs runtime of job - If
is_dynamic_au_onis true, we could have some sort of ML model that looks at the microbatch size and look at historical heuristics to figure out the best AU to apply to process the backlog most optimally (but don't overassign)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing AU assignment path and any telemetry or dynamic-AU entry points; the issue names no files or tests. Read how microbatch size, per-file row counts, runtime history, and the overassignment constraint are represented. Done means the dynamic path can use historical heuristics to select an AU for backlog processing without overassigning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- machine-learning, python
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100