prerequisite: auto save checkpoint

Motivation

Jobs fail at different stages.
The users will have to retrain their models from scratch if they fail to save the checkpoints / result files properly.
If they have a folder which will be saved automatically, it will be easier for users to manage these data.

Design

• two prerequisite
  • mount-autosave-folder: mount cluster storage to task container
  • auto-save: copy check-point folder to mount point
• cluster requirement:
  • cluster level storage
• schema

prerequisites:
  - type: data
    name: mount-autosave-folder
    plugin: com.microsoft.pai.runtimeplugin.cmd
    mountPoint: /mnt/auto-save
    callbacks:
       - event: taskStarts
        commands:
          - mount  cluster-data-path ${mountPoint} 
  - type: autosave
    requires: mount-autosave-folder
    name: autosave-results
    plugin: com.microsoft.pai.runtimeplugin.cmd
    path: /user/experiments/check-points
    callbacks:
       - event: taskStarts
        commands:
          - apt update 
          - apt install rsync
      - event: taskEnds
        commands:
          - rsync ${path} /mnt/auto-save

No files or tests are named. Start by tracing how prerequisites, runtime plugins, taskStarts, and taskEnds callbacks are handled, then inspect how cluster-level storage is configured. Done should support mounting cluster storage and copying the specified checkpoint or result path there when the task ends.