argoproj / argoproj/argo-workflows
Improve performance with parallelization when artifact compression is none
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
# Summary
When using compression of "none" on artifacts with 10000 files the get or put takes tens of minutes. Adjusting the resources on the executor doesn't seem to help. A quick look at pkg, and I believe this is because the s3client is going file by file sequentially instead of having a pool of workers running through the files in parallel.
# Use Cases
Uploading many files as an artifact for a subsequent highly parallel step, and also then joining the results after the step is complete.
---
**Message from the maintainers**:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
Contributor guide
Research direction
Start in pkg by tracing the s3client path used for artifact compression set to "none" and determine where files are transferred sequentially. Validate the change with an artifact containing many files and confirm that uploads and downloads complete faster through parallel work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cloud, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100