argoproj / argoproj/argo-workflows

Upload/download artifact directly from/to cloud storage: Add ability to do these tasks with artifact's metadata

Open
#9,879 2 comments 8 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

# Summary
Downloading/Uploading artifacts from/to S3 or any cloud-storage is working perfectly, but in some scenario I'd like to do that with some metadata

What change needs making?
- Add `metadata` field to `outputs.artifacts` e.g.
```yaml
outputs:
artifacts:
- name: test-data
path: >-
/tmp/test.data
metadata:
- name: mt-1
value: vl-1
- name: mt-2
value: {{=sprig.trim(workflow.parameters.some_value)}}
archive:
none:
{}
s3:
key: >-
path/to/file
region: >-
ca-central-1
bucket: >-
my-bucket
endpoint: s3.amazonaws.com
useSDKCreds: true

```
- Add `with-metadata` to `inputs.artifacts` as follow:
```yaml
inputs:
artifacts:
- name: test-data
path: >-
/tmp/test.data
with-metadata: "true"
archive:
none:
{}
s3:
key: >-
path/to/file
region: >-
ca-central-1
bucket: >-
my-bucket
endpoint: s3.amazonaws.com
useSDKCreds: true

```
- Add a new variable to `inputs.artifacts.test-data.metadata` to allow access to metadata of `test-data`.
# Use Cases

When would you use this?
Some scenario need more information from artifact's metadata rather than artifact's content. An ability to enrich data using metadata and pass them around argo workflow will add more flexibility on handling complex use cases.

---

**Message from the maintainers**:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or implementation entry points are named. Start by tracing artifact input/output handling and its S3 or cloud-storage paths; done means output metadata can be declared, input metadata can be requested, and the named artifact metadata variable is available.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.