argoproj / argoproj/argo-workflows

Improve performance with parallelization when artifact compression is none

Open
#4,014 11 comments 4 reactions 0 assignees View on GitHub
area/artifacts solution/workaround
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 20h
Merged PRs (30d)
138

Description

# Summary

When using compression of "none" on artifacts with 10000 files the get or put takes tens of minutes. Adjusting the resources on the executor doesn't seem to help. A quick look at pkg, and I believe this is because the s3client is going file by file sequentially instead of having a pool of workers running through the files in parallel.

# Use Cases

Uploading many files as an artifact for a subsequent highly parallel step, and also then joining the results after the step is complete.

---

**Message from the maintainers**:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

Contributor guide

Open the contributing guide

Research direction

Start in pkg by tracing the s3client path used for artifact compression set to "none" and determine where files are transferred sequentially. Validate the change with an artifact containing many files and confirm that uploads and downloads complete faster through parallel work.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cloud, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.