theupdateframework / theupdateframework/python-tuf

client/updater design: parallel downloads

Open
#1,158 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement ngclient
Dominant language
Python
Stars
1.7k
Forks
304
Avg merge
1d 2h
Merged PRs (30d)
17

Description

Updater should in some way support parallel downloads.

This came up because I found an obscure corner in pip that does index file downloads in parallel in pip (pip list --outdated): if I plug my TUF-support in there, TUF Updater gets called from multiple threads and very bad things happen.

Some notes on this:

  • for the pip use case, letting pip handle the actual downloads would remove 50% of the need for this -- but bin metadata downloads would still be sequential (and Updater.refresh() would still be thread-unsafe)
  • Still, parallel downloads (for both targets and bin metadata) is probably a wanted feature in other user-facing apps as well: it is faster and is something package managers already often do
  • this feature does not mean that TUF API absolutely needs to be threadsafe: many use cases would be covered by small API change: Updater.get_valid_targetinfos(list) and Updater.download_targets(list) could still be a single-threaded API but they could be implemented in a way that uses multiple threads and multiple connections to the server. I have no idea how easy this would be though.
  • The nicest solution might be a low-level Sans-IO style API (a state machine with events) or at least something that is asynchronous to the users. This API is then wrapped by a simple I/O layer that could look much like it now does. This would mean a large change.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the existing Updater.refresh(), Updater.get_valid_targetinfos(list), and Updater.download_targets(list) entry points, then review the pip parallel-download context described in the issue. Define a concrete API or concurrency scope, with completion meaning that targets and bin metadata can be downloaded in parallel without unsafe shared Updater state.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.