theupdateframework / theupdateframework/python-tuf

ngclient: support `StorageBackendInterface`?

Open
#2,676 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.7k
Forks
304
Avg merge
1d 2h
Merged PRs (30d)
17

Description

Description of issue or feature request:

Right now, tuf.ngclient is heavily tied to local system I/O: it assumes a metadata directory on disk that can be read/written. For example:

https://github.com/theupdateframework/python-tuf/blob/4d2ff8d37d30e94dbc0fe2cfa42bd46d2bb72414/tuf/ngclient/updater.py#L293-L312

This is problematic in distributed worker setups like Warehouse (PyPI), where each worker has its own container/entire VM and thus can't easily share on-disk TUF repos. In particular, this causes both reliability and security concerns:

  • Reliability: an unfortunate corruption in a single worker's TUF repo results in a hard-to-diagnose flaky worker, since each worker has its own copy of the repo.
  • Security: each worker's TUF repo is independently stored on a (machine-local) disk, making them harder to audit.

This problem was noted a few years back, before tuf.ngclient was created: https://github.com/theupdateframework/python-tuf/issues/1009. The solution then was to add a filesystem abstraction to the tuf.metadata APIs, which was done via https://github.com/secure-systems-lab/securesystemslib/pull/232 and https://github.com/theupdateframework/python-tuf/issues/1009. However, this abstraction wasn't added to the ngclient APIs, only to the low-level metadata ones.

Current behavior:

tuf.ngclient currently assumes that it can perform persistent local I/O for its repository.

Expected behavior:

tuf.ngclient should support an I/O abstraction (such as the pre-existing StorageBackendInterface, if suitable) for persistent repo operations, enabling use in distributed deployments.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tuf/ngclient/updater.py around lines 293-312 and compare its persistent repository I/O with the existing StorageBackendInterface used by the low-level metadata APIs. Determine the API and integration scope needed for ngclient, then verify that repository operations can use non-local storage in distributed deployments.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.