Formalize support for zstd compression: v1.1.0 ?

While reviewing https://github.com/moby/moby/pull/40820, I noticed that support for zstd was merged in master (proposal: https://github.com/opencontainers/image-spec/issues/787, implementation in https://github.com/opencontainers/image-spec/pull/788 and https://github.com/opencontainers/image-spec/pull/790), and some runtimes started implementing this;

• containerd: https://github.com/containerd/containerd/pull/3649
• containers/image: https://github.com/containers/image/pull/563
• (in progress) docker / moby: https://github.com/moby/moby/pull/40820 (currently for "extracting" only)

However, the current (v1.0.1) image-spec does not yet list zstd as a supported compression, which means that not all runtimes may support these images, and the ones that do are relying on a non-finalized specification, which limits interoperability (something that I think this specification was created for in the first place).

I think the current status is not desirable; not only does it limit interoperability (as mentioned), it will also cause complications Golang projects using this specification as a dependency; go modules will default to the latest tagged release, and some distributions (thinking of Debian) are quite strict about the use of unreleased versions. Golang project that want to support zstd would either have to "force" go mod to use a non-released version of the specification, or work around the issue by using a custom implementation (similar to the approach that containerd took: https://github.com/containerd/containerd/pull/3649).

In addition to the above, concerns were raised about the growing list of media-types (https://github.com/opencontainers/image-spec/issues/791), and suggestions were made to make this list more flexible.

The Image Manifest Property Descriptions, currently describes:

Implementations MUST support at least the following media types:

• application/vnd.oci.image.layer.v1.tar
• application/vnd.oci.image.layer.v1.tar+gzip
• application/vnd.oci.image.layer.nondistributable.v1.tar
• application/vnd.oci.image.layer.nondistributable.v1.tar+gzip

Followed by:

...An encountered mediaType that is unknown to the implementation MUST be ignored.

This part is a bit ambiguous (perhaps that's just my interpretation of it though);

• Should an implementation pull a manifest, and skip (ignore) layers with unknown compression, or should it produce an error?
• If the +zstd layer mediatype is not in the MUST list, is there any reason for including it in the list of OCI Media Types? After all, any media types not included in the list "could" be supported by an implementation, and must otherwise be ignored.

What's the way forward with this?

1. Tag current master as v1.1.0, only defining +zstd as a possible compression format for layers, but no requirement for implementations of the v1.1.0 specification to support them
2. Add the +zstd compression format to the list of required media types, and tag v1.1.0; projects implementing v1.1.0 of the specification MUST support zstd layers, or otherwise implement v1.0.x
3. Wait for the discussion about "generic" layer types (https://github.com/opencontainers/image-spec/issues/791, https://github.com/opencontainers/image-spec/issues/799) to be completed before tagging v1.1.0
4. Do a v1.1.0 release (1. or 2.), and leave 3. for a future (v1.2.0) release of the specification.

On a side-note, I noticed that the vnd.oci.image.manifest.v1+json was registered, but other mediatypes, including media-types for image layers are not; should they be?

Start by reading manifest.md and media-types.md at the referenced revision, then follow the linked image-spec issues 787, 791, and 799 and the related runtime pull requests. Done means the maintainers have resolved the zstd requirements, media-type guidance, and v1.1.0 release path in the specification.