mlcommons / mlcommons/mlcube_examples
Config 2.0 unified storage description
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 33
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
This proposes a unified storage description for config 2.0.
Today, MLCube relies on a simple "file path" approach to describe the inputs and outputs of their tasks. However, for many platforms, such like Kubernetes, it is not possible to use a single file path, because they either have complex storage backend, or use their own layer of storage abstractions, which do not use "paths" to refer to the corresponding locations in the data storage. This proposal aims to address this problem by providing a unified way of describing storage that can cover both local file systems and more complex storage solutions.
A storage backend can be described in the platform section of the config, which is supplied by the user at run-time. The storage description consists of 2 main parts: a name that will be used as a reference in the tasks' I/O paths, and a platform-specific spec that provides the details of the storage backend in the target platform, so that the runner can use it to find the right location of data.
We do not change the "path"-like descriptions of task inputs/outputs in order to keep that simple, however, we do introduce a "variable"-like component as a part of the path, so that we can use this "variable" as a reference to the corresponding storage backend and use the rest of the path as a relative path to the given storage.
A most straight-forward example of such "variable" is "$WORKSPACE" which is currently being used to refer to a specific dir in local file system. With the new proposal, the "$WORKSPACE", or any "$CUSTOM_NAME" defined by user, can refer to an arbitrary storage backend as specified in the platform section.
Since the detailed spec of the storage is specified in the platform part, it can be decoupled from the shared MLCube config and only appear in the user's config. This also means that how the spec of a given storage backend is writtern should be agreed between a user and a runner, and not relevant to the MLCube publisher.
While we do not have to provide a standard for that specs, we may provide some "guidelines/examples" for popular platforms so that there can be a convention for runner implementors.
The following is an example of how the storage backend can be defined, notice the specs in the platform section and how they are used in the tasks section. Notice also that if we give the storage a name of "WORKSPACE" then we may redirect our default workspace to the specified storage backend, without change the values in the task I/Os.
name: example-mlcube
platform:
storage:
- name: K8S_DATA
spec:
kubernetes:
pvc_name: my-pvc
- name: NFS_DATA
spec:
nfs:
host: 127.0.0.1
port: 2049
path: some/nfs/path
container:
image: mlcommons/mnist:0.0.1
build_context: "mnist"
build_file: "Dockerfile"
tasks:
download:
io:
- {name: data_dir, type: directory, io: output, default: $NFS_DATA/data}
- {name: log_dir, type: directory, io: output, default: $NFS_DATA/logs}
train:
io:
- {name: data_dir, type: directory, io: input, default: $K8S_DATA/data}
- {name: parameters_file, type: file, io: input, default: $K8S_DATA/parameters/default.parameters.yaml}
- {name: log_dir, type: directory, io: output, default: $K8S_DATA/logs}
- {name: model_dir, type: directory, io: output, default: $K8S_DATA/model}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the config schema and runner entry points that handle the platform and tasks sections; the issue names no files or tests. Read how task I/O paths and the $WORKSPACE variable are currently resolved. Done means platform.storage definitions can describe backend-specific specs and task paths can reference them consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- yaml
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100