argoproj / argoproj/argo-workflows
Upload/download artifact directly from/to cloud storage: Add ability to do these tasks with artifact's metadata
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
# Summary
Downloading/Uploading artifacts from/to S3 or any cloud-storage is working perfectly, but in some scenario I'd like to do that with some metadata
What change needs making?
- Add `metadata` field to `outputs.artifacts` e.g.
```yaml
outputs:
artifacts:
- name: test-data
path: >-
/tmp/test.data
metadata:
- name: mt-1
value: vl-1
- name: mt-2
value: {{=sprig.trim(workflow.parameters.some_value)}}
archive:
none:
{}
s3:
key: >-
path/to/file
region: >-
ca-central-1
bucket: >-
my-bucket
endpoint: s3.amazonaws.com
useSDKCreds: true
```
- Add `with-metadata` to `inputs.artifacts` as follow:
```yaml
inputs:
artifacts:
- name: test-data
path: >-
/tmp/test.data
with-metadata: "true"
archive:
none:
{}
s3:
key: >-
path/to/file
region: >-
ca-central-1
bucket: >-
my-bucket
endpoint: s3.amazonaws.com
useSDKCreds: true
```
- Add a new variable to `inputs.artifacts.test-data.metadata` to allow access to metadata of `test-data`.
# Use Cases
When would you use this?
Some scenario need more information from artifact's metadata rather than artifact's content. An ability to enrich data using metadata and pass them around argo workflow will add more flexibility on handling complex use cases.
---
**Message from the maintainers**:
Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.
Contributor guide
Research direction
No files, tests, or implementation entry points are named. Start by tracing artifact input/output handling and its S3 or cloud-storage paths; done means output metadata can be declared, input metadata can be requested, and the named artifact metadata variable is available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100