argoproj / argoproj/argo-workflows

Enable Custom Metadata for S3 Output Artifacts

Open
#13,488 0 comments 3 reactions 0 assignees View on GitHub
area/artifacts
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

# Summary

I propose adding a feature to Argo Workflows that allows setting object metadata (user metadata, headers like content-disposition, content-encoding, etc) for output artifacts being stored on S3-compatible storage. The `minio-go` sdk used by Argo [already supports these attributes in `PutObject`](https://min.io/docs/minio/linux/developers/go/API.html#PutObject:~:text=multipart%20upload%20operation.-,minio.PutObjectOptions,-Field)

## Use Cases

This feature would be particularly useful in the following scenarios:

1. **Compliance and Security**: Custom metadata can be used to store information related to data classification, compliance, or security requirements.

2. **Data Management**: Users can store versioning information, timestamps, or other relevant data in metadata, improving data management and retrieval processes.

3. **Integration with Other Systems**: When integrating with other systems that rely on specific metadata keys, this feature would facilitate smoother interoperability.

4. **Tagging and Categorization**: Users can tag or categorize artifacts with metadata, aiding in organization and searchability within large datasets.

---

**Message from the maintainers**:

Love this feature request? Give it a 👍. We prioritise the proposals with the most 👍.

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files, tests, or configuration entry points. Start by locating the S3 output-artifact upload path and the minio-go PutObject call, then determine how custom metadata and headers should be configured; done means S3 artifacts accept and preserve the requested metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.