mlcommons / mlcommons/storage

Enforce --data-dir post-YAML for all training commands; eliminate inherited DLIO YAML default

Open
#508 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Future
Dominant language
Python
Stars
203
Forks
67
Avg merge
20m
Merged PRs (30d)
8

Description

Summary

Ensure --data-dir for training cannot be silently defaulted. Today the argparse layer has no default, but enforcement has gaps and a hardcoded value living in DLIO workload YAMLs acts as an effective inherited default when no override is supplied. The benchmark should fail loudly any time the user has not explicitly provided --data-dir via the CLI or a YAML config file.

Current state

Layer Behaviour Status
argparse (mlpstorage_py/cli/training_args.py:171-178) No default= kwarg; default is None. ✅ Already correct
File-mode enforcement (mlpstorage_py/cli_parser.py:167-175) parser.error() fires before apply_yaml_config_overrides. A user supplying data_dir: via --config-file for file mode is rejected before YAML has a chance to populate it. ❌ Bug
Object-mode enforcement (mlpstorage_py/cli/training_args.py:269-276) sys.exit(EXIT_CODE.INVALID_ARGUMENTS) after YAML. ✅ Works
datasize command Excluded from the enforcement check at cli_parser.py:169. No fallback elsewhere. ❌ Slips through
Env-var fallback No MLPS_DATA_DIR / MLPSTORAGE_DATA_DIR exists in the codebase. (Doesn't exist; intentionally out of scope)
DLIO workload YAMLsconfigs/dlio/workload/*.yaml Hardcode dataset.data_folder: data/<model>/ (e.g. data/unet3d/, data/resnet50, data/dlrm/, data/flux/). mlpstorage_py/benchmarks/dlio.py:410 only calls add_datadir_param() when self.args.data_dir is truthy, so when --data-dir is unset and enforcement is skipped (datasize, or any escape route), DLIO silently uses this relative-path default. ❌ Silent inherited default

Proposal

Single post-YAML check in validate_training_arguments (or the dispatcher) that fires for every training command (datasize, datagen, run, configview) regardless of data_access_protocol. Remove the early file-mode check in cli_parser.py so YAML config files can populate data_dir for either protocol. Error message should be actionable and point to both the CLI flag and the --config-file YAML key:

ERROR: --data-dir is required for training <command>.
  Specify --data-dir <path> on the command line, or set 'data_dir:' in the file
  passed via --config-file. There is no fallback default — the DLIO workload
  YAML's data_folder will not be used.

Out of scope (deferred)

  • Verifying that the effective --data-dir actually lives on the system under test. Ideal — currently a user can point at a path that doesn't exist on the SUT and not notice until the benchmark stalls. The ideal check would stat() the path (or HEAD the object prefix) on each MPI rank's host and refuse to start otherwise. Decision: too much codebase disruption at this stage of the release cycle; record here so it isn't lost.
  • Adding a MLPS_DATA_DIR env-var fallback. Out of scope for this issue per the requesting user: the goal is to plug the silent-default leak, not expand the parameter-setting surface.
  • Clearing data_folder from the DLIO workload YAMLs. Higher blast radius (affects anyone running DLIO directly), and the proposed enforcement makes the YAML default unreachable from the mlpstorage CLI anyway, so not strictly necessary.

Test plan

  • Unset --data-dir, file mode, all four commands → fail with the new message (datasize currently slips through; this is the main regression class to cover).
  • Unset --data-dir, object mode, all four commands → fail with the new message.
  • --data-dir on CLI, all commands × both protocols → pass.
  • --data-dir only in --config-file YAML, all commands × both protocols → pass (currently broken for file mode).
  • Confirm dlio.py:410's add_datadir_param() is reached on every training command after the fix.

Files in scope

  • mlpstorage_py/cli_parser.py — remove the early file-mode block at L167–175.
  • mlpstorage_py/cli/training_args.py — extend the existing validate_training_arguments check to cover all commands and both protocols.
  • mlpstorage_py/cli/help_formatter.py — already lists --data-dir under "Required:" for the relevant commands; no change needed.
  • Tests: tests/unit/test_cli.py, tests/unit/test_cli_parser.py, tests/unit/test_parser_modes.py — add cases for the previously-uncovered datasize and --config-file-file-mode paths.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with validate_training_arguments in mlpstorage_py/cli/training_args.py and the early file-mode check in mlpstorage_py/cli_parser.py, then inspect the DLIO call at mlpstorage_py/benchmarks/dlio.py:410. Run the relevant CLI tests in tests/unit/test_cli.py, tests/unit/test_cli_parser.py, and tests/unit/test_parser_modes.py. Done means every listed command rejects a missing data_dir while CLI and YAML values work for both protocols.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.