argoproj / argoproj/argo-workflows
Capture artifact meta-data
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
**Is this a BUG REPORT or FEATURE REQUEST?**:
FEATURE REQUEST
**What happened**:
I would like to specify custom metadata associated with the output artifacts in workflows spec
**What you expected to happen**:
It would be great if we could specify metadata in the s3 part of output artifacts spec (https://github.com/argoproj/argo/blob/11e57f4dea93fde60b204a5e7675fec999c66f56/examples/output-artifact-s3.yaml). Something like this:
```
s3:
# Use the corresponding endpoint depending on your S3 provider:
# AWS: s3.amazonaws.com
# GCS: storage.googleapis.com
# Minio: my-minio-endpoint.default:9000
endpoint: s3.amazonaws.com
bucket: my-bucket
# NOTE: by default, output artifacts are automatically tarred and gzipped before saving.
# As a best practice, .tgz or .tar.gz should be suffixed into the key name so the
# resulting object has an accurate file extension and mime-type. If archive is set to
# 'none', then preserve the appropriate file extension for the key name
key: path/in/bucket/hello_world.txt.tgz
metadatas:
- name: custom-field
valueFrom:
path: /tmp/custom-field.txt
- name: workflow-type
value: s3-workflow
# accessKeySecret and secretKeySecret are secret selectors. It references the k8s secret
# named 'my-s3-credentials'. This secret is expected to have have the keys 'accessKey'
# and 'secretKey', containing the base64 encoded credentials to the bucket.
accessKeySecret:
name: my-s3-credentials
key: accessKey
secretKeySecret:
name: my-s3-credentials
key: secretKey
```
**Anything else we need to know?**:
https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
This is how I would do this using minio client (https://docs.min.io/docs/minio-client-complete-guide) :
```
mc cp --attr key1=value1,key2=value2 myobject.txt play/mybucket
```
**Environment**:
- Argo version:
```
argo: v2.3.0
BuildDate: 2019-05-20T22:11:09Z
GitCommit: 88fcc70dcf6e60697e6716edc7464a403c49b27e
GitTreeState: clean
GitTag: v2.3.0
GoVersion: go1.11.5
Compiler: gc
Platform: linux/amd64
```
Thanks
Contributor guide
Research direction
Start with examples/output-artifact-s3.yaml and the workflow output-artifact S3 specification. Trace how the S3 artifact settings are handled, including values read from paths, and verify the resulting object metadata against the requested fields. Done means workflow authors can specify metadata for S3 output artifacts and the metadata is preserved on upload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100