google / google/go-containerregistry

FR: crane upload

Open
#934 6 comments 2 reactions 1 assignee Claimed by @imjasonh View on GitHub
lifecycle/frozen
Dominant language
Go
Stars
4k
Forks
686
Avg merge
2d 12h
Merged PRs (30d)
26

Description

I'd like a new subcommand that uploads blobs.

I need to write up more about this, but `crane` hits a really sweet spot for most things where we're doing as little as possible to implement container-specific details, then getting out of the way. `crane export` and `crane auth` are great examples of this. So are `crane {manifest,blob,ls,catalog,delete,digest,tag}`.

There are also higher level things that are often useful, but I really like the lower-level things, because they make it possible to do things as a bash one-liner that otherwise require loading up an IDE and pulling in a ton of dependencies.

Uploading blobs is a huge missing piece.

## strawman

```
$ crane upload reg.example.com/my-repo < some-file
sha256:db98fc6f11f08950985a203e07755c3262c680d00084f601e7304b768c83b3b1
```

The output is the digest of the uploaded thing.

## output

Several modes of output would be useful:

A mode that prints the digest as a ref (maybe `-q` for "qualify?"), for use with `crane blob` or perhaps `crane append`:

```
$ crane upload -q reg.example.com/my-repo < some-file
reg.example.com/my-repo@sha256:db98fc6f11f08950985a203e07755c3262c680d00084f601e7304b768c83b3b1

# Should be a no-op (assuming this doesn't overwrite some-file before we read it):
$ crane blob $(crane upload -q reg.example.com/my-repo < some-file) > some-file
```

A mode that prints a descriptor, which should include the size:
```
$ crane upload reg.example.com/my-repo < some-file | jq .
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 843,
"digest": "sha256:db98fc6f11f08950985a203e07755c3262c680d00084f601e7304b768c83b3b1"
}
```

(Note the `| jq .` to pretty-print it -- this should be one line of output, normally.)

## flags

There is plenty of opportunity for flags and bikeshedding of those flags.

For example, should compression be on or off by default? Given that it's possible to do this:

```
$ gzip some-file -c | crane upload reg.example.com/my-repo
```

I don't see a reason for it? Perhaps a `-z` flag as a convenience, but if you want to set a specific compression level, that's up to you.

## unsolved

### diffid

I'm not sure how to represent diffid here, for things that get compressed.

The problem is two-fold:
1. We don't really have a way to persist metadata like this. If we want to be able to do something interesting with a blob, it would be nice if we didn't have to download the whole thing, check to see if it's gzipped, and hash it to compute the diffid.
2. How do we actually expose this in a composable way?

Some ideas...

If we are printing a descriptor _and_ we're compressing it, add the diffid as an annotation. We can have other tools understand this annotation when reconstructing a config file (like in `pkg/v1/mutate`).

```
$ crane upload -z reg.example.com/my-repo < some-file | jq .
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 843,
"digest": "sha256:db98fc6f11f08950985a203e07755c3262c680d00084f601e7304b768c83b3b1",
"annotations": {
"dev.ggcr.crane.diffid": "sha256:dbf2c0f42a39b60301f6d3936f7f8adb59bb97d31ec11cc4a049ce81155fef89"
}
}
```

If we're not printing a descriptor, we could just output two lines, where the first is digest and the second diffid:
```
$ crane upload reg.example.com/my-repo -z < some-file
sha256:db98fc6f11f08950985a203e07755c3262c680d00084f601e7304b768c83b3b1
sha256:dbf2c0f42a39b60301f6d3936f7f8adb59bb97d31ec11cc4a049ce81155fef89
```

Or we could have an output file?
```
$ crane upload --diffid some-file-diffid.sha256 reg.example.com/my-repo -z < some-file
sha256:db98fc6f11f08950985a203e07755c3262c680d00084f601e7304b768c83b3b1
```

Or just tell people to manually run `sha256sum` before uploading? That seems reasonable enough to me...

### media type

If we're printing a descriptor, I think it's safe to assume a default of the docker layer media types, compressed or not (we should probably sniff the contents to see if it is).

However, we might want to set an arbitrary media type, so that should perhaps just be a flag? Let's say I want to upload some json. The registry doesn't care here, actually, but if we want to compose this with another tool, it would be useful to have an accurate descriptor.

```
$ crane upload -m "application/json" reg.example.com/my-repo < config.json | jq .
{
"mediaType": "application/json",
"size": 809,
"digest": "sha256:246c399c7f7b05e2a241ce0771456bee9eaa61d5015997237c920d69fd024443"
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.