Enforce --data-dir post-YAML for all training commands; eliminate inherited DLIO YAML default
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 203
- Forks
- 67
- Avg merge
- 20m
- Merged PRs (30d)
- 8
Description
Summary
Ensure --data-dir for training cannot be silently defaulted. Today the argparse layer has no default, but enforcement has gaps and a hardcoded value living in DLIO workload YAMLs acts as an effective inherited default when no override is supplied. The benchmark should fail loudly any time the user has not explicitly provided --data-dir via the CLI or a YAML config file.
Current state
| Layer | Behaviour | Status |
|---|---|---|
argparse (mlpstorage_py/cli/training_args.py:171-178) |
No default= kwarg; default is None. |
✅ Already correct |
File-mode enforcement (mlpstorage_py/cli_parser.py:167-175) |
parser.error() fires before apply_yaml_config_overrides. A user supplying data_dir: via --config-file for file mode is rejected before YAML has a chance to populate it. |
❌ Bug |
Object-mode enforcement (mlpstorage_py/cli/training_args.py:269-276) |
sys.exit(EXIT_CODE.INVALID_ARGUMENTS) after YAML. |
✅ Works |
datasize command |
Excluded from the enforcement check at cli_parser.py:169. No fallback elsewhere. |
❌ Slips through |
| Env-var fallback | No MLPS_DATA_DIR / MLPSTORAGE_DATA_DIR exists in the codebase. |
(Doesn't exist; intentionally out of scope) |
DLIO workload YAMLs — configs/dlio/workload/*.yaml |
Hardcode dataset.data_folder: data/<model>/ (e.g. data/unet3d/, data/resnet50, data/dlrm/, data/flux/). mlpstorage_py/benchmarks/dlio.py:410 only calls add_datadir_param() when self.args.data_dir is truthy, so when --data-dir is unset and enforcement is skipped (datasize, or any escape route), DLIO silently uses this relative-path default. |
❌ Silent inherited default |
Proposal
Single post-YAML check in validate_training_arguments (or the dispatcher) that fires for every training command (datasize, datagen, run, configview) regardless of data_access_protocol. Remove the early file-mode check in cli_parser.py so YAML config files can populate data_dir for either protocol. Error message should be actionable and point to both the CLI flag and the --config-file YAML key:
ERROR: --data-dir is required for training <command>.
Specify --data-dir <path> on the command line, or set 'data_dir:' in the file
passed via --config-file. There is no fallback default — the DLIO workload
YAML's data_folder will not be used.
Out of scope (deferred)
- Verifying that the effective
--data-diractually lives on the system under test. Ideal — currently a user can point at a path that doesn't exist on the SUT and not notice until the benchmark stalls. The ideal check wouldstat()the path (or HEAD the object prefix) on each MPI rank's host and refuse to start otherwise. Decision: too much codebase disruption at this stage of the release cycle; record here so it isn't lost. - Adding a
MLPS_DATA_DIRenv-var fallback. Out of scope for this issue per the requesting user: the goal is to plug the silent-default leak, not expand the parameter-setting surface. - Clearing
data_folderfrom the DLIO workload YAMLs. Higher blast radius (affects anyone running DLIO directly), and the proposed enforcement makes the YAML default unreachable from the mlpstorage CLI anyway, so not strictly necessary.
Test plan
- Unset
--data-dir, file mode, all four commands → fail with the new message (datasize currently slips through; this is the main regression class to cover). - Unset
--data-dir, object mode, all four commands → fail with the new message. --data-diron CLI, all commands × both protocols → pass.--data-dironly in--config-fileYAML, all commands × both protocols → pass (currently broken for file mode).- Confirm
dlio.py:410'sadd_datadir_param()is reached on every training command after the fix.
Files in scope
mlpstorage_py/cli_parser.py— remove the early file-mode block at L167–175.mlpstorage_py/cli/training_args.py— extend the existingvalidate_training_argumentscheck to cover all commands and both protocols.mlpstorage_py/cli/help_formatter.py— already lists--data-dirunder "Required:" for the relevant commands; no change needed.- Tests:
tests/unit/test_cli.py,tests/unit/test_cli_parser.py,tests/unit/test_parser_modes.py— add cases for the previously-uncovered datasize and--config-file-file-mode paths.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with validate_training_arguments in mlpstorage_py/cli/training_args.py and the early file-mode check in mlpstorage_py/cli_parser.py, then inspect the DLIO call at mlpstorage_py/benchmarks/dlio.py:410. Run the relevant CLI tests in tests/unit/test_cli.py, tests/unit/test_cli_parser.py, and tests/unit/test_parser_modes.py. Done means every listed command rejects a missing data_dir while CLI and YAML values work for both protocols.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100