[V4] Simplify the set of controls around downloading artifacts
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 598
- Forks
- 168
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 86
Description
Is your feature request related to a problem? Please describe.
There exist many different controls around content downloading that all accomplish very similar, but not exactly the same things.
MAX_CONCURRENT_CONTENT
download_concurrency (on Remote)
rate_limit (on Remote)
SYNC_MAX_IN_FLIGHT_ITEMS and SYNC_MAX_IN_FLIGHT_MB (recently added)
Batch size of the download stage
Having all of these knobs is confusing and makes the code much more complicated than it needs to be.
We should audit all the different ways in which these overlap and try to simplify if we can.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
derived from this conversation: https://github.com/pulp/pulpcore/pull/7649#discussion_r3209269413
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by auditing MAX_CONCURRENT_CONTENT, Remote.download_concurrency, Remote.rate_limit, SYNC_MAX_IN_FLIGHT_ITEMS, SYNC_MAX_IN_FLIGHT_MB, and the download-stage batch size. Read the linked pull request discussion for context and trace where each control is applied. Done means the overlapping controls have a simpler, agreed design with their responsibilities clearly defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100