Storage orchestration

Today swarmkit supports persistent storage via the Mounts API. This enables users to either pre-create some bit of storage and then mount it into the service, or have the service create the storage on the node when the task is scheduled (when provided the correct options).

As part of the Docker 1.13 release, support for templating was added which can kinda-sorta make a service orchestrate this a bit better by incorporating things like .Task.Slot into the volume name such that any time a task in a particular slot is created it would (hopefully) get the same volume back, but slots would not share a volume specifically.

This is all to basically work around the fact that we don't have cluster-aware storage.
Here I'd like to discuss what the API's for cluster-aware storage may look like.
What I'd like to do is get away from the user manually attaching volumes to a service (for the general case) and let the service object handle the heavy lifting.

For storage there are multiple use-cases as far as service-level orchestration goes:

1. Storage which is shared by all tasks of a service (e.g. --mount type=volume,src=my_data,target=/data)
2. Storage which is essentially throw-away per task (e.g. --mount type=volume,target=/data)
3. Storage which is dedicated per slot (no e.g., close but not really: --mount type=volume,source=data-{{.Task.Slot}},target=/data)

Data retention is also something to deal with. I think the default here would be to retain all data, but at least provide a way for data to be automatically cleaned up if requested.

For actual creation of storage objects, we really don't want the API's to be too tied to specific drivers... e.g. I shouldn't be telling the persistence API to use the ebs driver since this is completely non-portable.
Instead we should constrain creation based on other items. Kub has storage classes, which we could mimic, but also could just use labels for this.

Finally if the storage provided to the container should be pre-populated with some data. Maybe a snapshot/clone... maybe data from the image (docker default), or just empty storage.

Here's a rough idea of what this might look like in a compose file:

services:
    acme:
        image: acme-image
        persistence:
          - target: /var/acme
            # service = single volume for all replicas
            # slot = each slot has a unique volume, If a task dies, the same volume would be re-used for the replacement task
            # task = each container task gets its own unique volume
            level: {service,slot,task}
            cleanup:
              type: {never,on-remove,deferred} # default never
              deferred: # deferred options
                after: {duration}
            # should we just use labels here? e.g., label driver with `class=platinum` and constrain here based on labels.class=platinum
            class: {bronze, silver, gold, platinum, <whatever>}
            # alternative:
            constraints:
              label:
                - "class=bronze"
            populate:
              # one of
              volume:
                # one of... if/when these things actually exist
                snapshot: <id or name>
                clone: <id or name>
                name: <id or name>
              image:
                path: /seed-data

No files, tests, or entry points are named. Start by reviewing the existing Mounts API and the service/task scheduling model described here. Done means reaching agreement on a cluster-aware persistence API covering sharing levels, retention, placement constraints, and data population without binding it to a specific storage driver.