nextflow-io / nextflow-io/nextflow
Restrict azcopy memory usage by task setting
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
New feature
Sometimes azcopy can use more memory than is available. See https://github.com/nextflow-io/nextflow/pull/5799#issuecomment-2939233221 for a description.
Use case
We can set the maximum azcopy memory usage so that it does not exhaust local memory and crash.
Suggested implementation
We should be able to add the following environment variable in the same manner as other azcopy opts: AZCOPY_BUFFER_GB https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-optimize#optimize-memory-use
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named. Start by locating where other azcopy options are added and how task settings become environment variables; then trace the azcopy task entry point. Done means a task setting controls AZCOPY_BUFFER_GB, with coverage for the resulting environment variable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, groovy
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100