Make training initialization and runtime services consume configs
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
Parent initiative: #3554
Depends on #7324 (explicit configuration preparation and initialization phases).
## Goal
Make the initialization path consume `PretrainConfigContainer` and its subconfigs so CLI-built and directly constructed configs initialize equivalent runtime services without relying on global args in the migrated helpers.
## Scope
- Migrate the initialization helpers extracted in #7324 and relevant `initialize.py`/`global_vars.py` consumers to configs or explicit config-derived inputs.
- Cover tokenizer construction and vocabulary resolution, distributed initialization, RNG, microbatch-calculator setup, logging/timers, profiling setup, rerun/straggler/fault-tolerance services, and associated async-checkpoint-worker setup.
- Complete the args-to-config mappings needed by these consumers. Resolve the fault-tolerance config schema against the parent initiative's mapping before implementing it; existing fault injection settings and FT/autoresume controls are not interchangeable.
- Move the validation and dependent-field derivation needed for initialization into appropriate config/shared helpers, with one implementation used by both input paths.
- Separate validation of requested launch topology before distributed setup from querying actual process-group sizes afterward. Replace runtime-derived args fields with helpers only where those helpers are valid at that lifecycle stage.
- Preserve tokenizer/vocabulary precedence, early monitoring, service ordering, restart, and cleanup behavior established by #7324.
- Keep compatibility args access localized to explicitly identified legacy entrypoint boundaries; migrated initialization helpers must not silently fall back to global args.
Model/DDP/optimizer construction and full-loop config migration follow in separate work. Shared initialization rules must remain consistent with those consumers during the transition.
## Completion criteria
- Initialization can be exercised from directly constructed configs with global `get_args()` unavailable in the migrated path.
- CLI and direct-config paths produce equivalent resolved tokenizer/vocabulary, runtime-service inputs, distributed topology, RNG initialization, and microbatch setup.
- Tests cover relevant translation rules, invalid combinations, pre/post-distributed derivation, startup ordering, and cleanup/reinitialization.
- Representative original MLM versus refactored MLM runs preserve initialization behavior and affected training behavior.
- Any remaining compatibility boundary is explicit and does not act as a second configuration authority.
## Out of scope
- Claiming the entire training loop is independent of args after this PR.
- Bulk migration of unrelated validation or model/optimizer/runtime consumers.
- `TrainState` ownership (#3555), new checkpoint metadata, callback APIs, or PEFT behavior.
Contributor guide
Research direction
Start by reading the dependent configuration work in #7324, then inspect the initialization helpers and consumers in initialize.py and global_vars.py. Trace how CLI-built and directly constructed configs currently reach tokenizer, distributed, RNG, microbatch, logging, profiling, and fault-tolerance setup. Done means the completion criteria hold, including equivalent paths, covered validation and lifecycle cases, preserved ordering and cleanup, and no implicit global-args fallback in migrated helpers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems, machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100