[User Story] Dataset: integrate data prerequisite into marketplace and job submission page

Motivation

#5145 has extended the prerequisite field. But users can only use and share prerequisites in job yaml. We can support UI for prerequistes, especially for data prerequisite. This issue will explain how the users create and use a data prerequisite in the cluster. With this feature, cluster users can easily share datasets with each other, and it may benefit future features e.g. dataset caching and optimization.

Explanation

How do users create a dataset in the cluster?

Dataset item that doesn't need a PVC storage

The user should create a dataset item in marketplace. dataset item has a prerequisite spec and other misc info (e.g. title, usage) in marketplace.

If the dataset is just downloaded from the Internet, it should have the following spec:

name: mnist
type: data
plugin: com.microsoft.pai.runtimeplugin.cmd
callbacks:
- event: taskStarts
   commands:
    - wget "<.....>" -O /dataset/mnist/<...>

Dataset item that needs a PVC storage

If the dataset is already saved in a PVC, it should have the following spec:

name: imagenet
type: data
plugin: com.microsoft.pai.runtimeplugin.cmd
requireStorages:
- name: confignfs
  mountPath: /mnt/confignfs
callbacks:
- event: taskStarts
   commands:
    - ln -s "/dataset/imagenet" "/mnt/confignfs/users/mine/presaved-imagenet"

Here we define a new field: requireStorages. It shares the same spec as the current implementation. If this prerequisite is included in a job, we should merge the storage field here with other PVC storage.

How do users use dataset in the cluster?

On marketplace pages

On marketplace pages, users can click use to create an empty job with the corresponding dataset.

On job submission page

On job submission page, users can select his/her dataset by the field under taskrole section.

How to represent marketplace prerequisite in job yaml?

The dataset prerequisite from marketplace will be expressed as marketplace://prerequisites/itemId/<item-id>

One example is as follows:

taskRoles:
  taskrole:
    prerequisites: ["marketplace://prerequisites/itemId/1"]
    instances: 1
    completion:
      minFailedInstances: 1
    taskRetryCount: 0
    dockerImage: docker_image_0
    resourcePerInstance:
      gpu: 1
      cpu: 3
      memoryMB: 29065
    commands:
      - echo 1

The webportal page should provide a link to marketplace for the user.

After submission, rest-server will parse these marketplace items and pass them to db controller and runtime. Rest-server should also take care of requireStorages, and merge it with other storage spec carefully.

The following errors can happen in rest-server:

• The user does not have permission to requireStorages.
  • Do we need to hide these datasets for users? Currently it is hard to implement. Maybe left to future work.
• The corresponding prerequisite can not be found.
• Fail to call marketplace's API.
• Fail to download the prerequisite item.

Other features

We can enable urls like http(s):// in addition to marketplace://. It will bring a lot of convenience and easy to implement.

taskRoles:
  taskrole:
    prerequisites: ["https://raw.githubusercontent.com/microsoft/pai/master/contrib/xxxx.yml"]
    instances: 1
    completion:
      minFailedInstances: 1
    taskRetryCount: 0
    dockerImage: docker_image_0
    resourcePerInstance:
      gpu: 1
      cpu: 3
      memoryMB: 29065
    commands:
      - echo 1

Implementation

• marketplace: provide backend and ui
• add field requireStorages
• webportal: ui change and validation
• rest-server: validation, and parse prerequisites
• database-controller
• runtime: run prerequisites

The issue spans the marketplace, webportal, rest-server, database-controller, and runtime entry points listed in its implementation checklist. Start by tracing how prerequisites currently flow from job YAML through submission and storage handling. Done means the marketplace and web UI support dataset selection, rest-server validates and merges requireStorages, and the downstream components execute the prerequisite safely.