S3 sync recursively with per-object metadata
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
I'm looking to take advantage of the `aws s3 sync` command, but provide per-object metadata (i.e. metadata that can change per object) rather than provide global metadata with `--metadata`.
Right now, I have basically a couple of options:
- Write my own syncing procedures. This would seem duplicative of the work/testing that has gone into the `sync` command.
- Include metadata in separate objects, e.g. /path/to/object and /path/to/object.meta. This isn't the greatest, means I have to pay extra and also manage the metadata within my application.
- Upload on a per-object basis. This isn't going to be performant for my use case.
What would be nice is if I could somehow indicate to the CLI that I want to map each object to a set of metadata, and then upload each object with that metadata. A couple of solutions come to mind:
- A giant JSON file mapping each key name to a metadata hash e.g.:
``` json
{
"path/to/object1": {"key1": "value1", "key2": "value2},
...etc...
}
```
``` shell
$ aws s3 sync /some/dir s3://somebucket --metadata-mapping /path/to/meta/mapping.json
```
- Some convention for writing metadata locally into a separate file per intended object, and having the `sync` command read the metadata for each object prior to uploading. For example, I could have a local directory:
``` shell
$ ls /path/to/local/files
file1
file1.meta
$ cat file1.meta
{
"key1": "value1",
"key2": "value2"
}
$ aws s3 sync /path/to/local/files s3://somebucket --object-metadata '$filename.meta'
```
(So when this is run, the `$filename.meta` files would just be read for metadata, and would not be transferred)
- A callback that takes the local filename as a parameter and spits out the metadata, e.g.
``` shell
$ ls /path/to/local/files
file1
$ lookup-metadata.py /path/to/local/files/file1
{
"key1": "value1"
}
$ aws s3 sync /path/to/local/files s3://somebucket --metadata-callback lookup-metadata.py
```
Alternatively, what would be really great is if the syncing functionality were available independently of the CLI from within Python (without requiring me to figure out the internals of how to properly initialize the CLI environment, etc.), so that I could subclass and customize the process. I started going down this route somewhat, but am worried that this API is not for public consumption and would break in the future.
Any thoughts?
Contributor guide
Assessment
This issue has not been assessed yet.