argoproj / argoproj/argo-workflows

Capture artifact meta-data

Open
#1,459 0 comments 1 reaction 0 assignees View on GitHub
area/artifacts
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

**Is this a BUG REPORT or FEATURE REQUEST?**:

FEATURE REQUEST

**What happened**:

I would like to specify custom metadata associated with the output artifacts in workflows spec

**What you expected to happen**:

It would be great if we could specify metadata in the s3 part of output artifacts spec (https://github.com/argoproj/argo/blob/11e57f4dea93fde60b204a5e7675fec999c66f56/examples/output-artifact-s3.yaml). Something like this:
```
s3:
# Use the corresponding endpoint depending on your S3 provider:
# AWS: s3.amazonaws.com
# GCS: storage.googleapis.com
# Minio: my-minio-endpoint.default:9000
endpoint: s3.amazonaws.com
bucket: my-bucket

# NOTE: by default, output artifacts are automatically tarred and gzipped before saving.
# As a best practice, .tgz or .tar.gz should be suffixed into the key name so the
# resulting object has an accurate file extension and mime-type. If archive is set to
# 'none', then preserve the appropriate file extension for the key name
key: path/in/bucket/hello_world.txt.tgz

metadatas:
- name: custom-field
valueFrom:
path: /tmp/custom-field.txt
- name: workflow-type
value: s3-workflow

# accessKeySecret and secretKeySecret are secret selectors. It references the k8s secret
# named 'my-s3-credentials'. This secret is expected to have have the keys 'accessKey'
# and 'secretKey', containing the base64 encoded credentials to the bucket.
accessKeySecret:
name: my-s3-credentials
key: accessKey
secretKeySecret:
name: my-s3-credentials
key: secretKey
```

**Anything else we need to know?**:

https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

This is how I would do this using minio client (https://docs.min.io/docs/minio-client-complete-guide) :
```
mc cp --attr key1=value1,key2=value2 myobject.txt play/mybucket
```

**Environment**:
- Argo version:
```
argo: v2.3.0
BuildDate: 2019-05-20T22:11:09Z
GitCommit: 88fcc70dcf6e60697e6716edc7464a403c49b27e
GitTreeState: clean
GitTag: v2.3.0
GoVersion: go1.11.5
Compiler: gc
Platform: linux/amd64
```

Thanks

Contributor guide

Open the contributing guide

Research direction

Start with examples/output-artifact-s3.yaml and the workflow output-artifact S3 specification. Trace how the S3 artifact settings are handled, including values read from paths, and verify the resulting object metadata against the requested fields. Done means workflow authors can specify metadata for S3 output artifacts and the metadata is preserved on upload.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.