tuna / tuna/tunasync-scripts

Add support for syncing CEP-16 sharded repodata for conda-forge

Open
#207 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
495
Forks
178
PR merge metrics
No merged PRs in 30d

Description

Hi, I noticed that the TUNA Anaconda mirror does not seem to provide CEP-16 sharded repodata for conda-forge, while the upstream conda-forge channel already provides it.

For example, upstream conda-forge has:

curl -I https://conda.anaconda.org/conda-forge/win-64/repodata_shards.msgpack.zst

which returns:

HTTP/1.1 200 OK
Content-Type: binary/octet-stream
Content-Length: 414167

and:

curl -I https://conda.anaconda.org/conda-forge/noarch/repodata_shards.msgpack.zst

also returns:

HTTP/1.1 200 OK
Content-Type: binary/octet-stream
Content-Length: 929729

However, the TUNA mirror currently returns 404 for the same file:

curl -I https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata_shards.msgpack.zst

returns:

HTTP/1.1 404 Not Found

while the traditional flat repodata file is available:

curl -I https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.json

returns:

HTTP/1.1 200 OK

This causes recent versions of mamba / libmamba to print warnings such as:

Shard Index for conda-forge/noarch not available, falling back to flat repodata
Shard Index for conda-forge/win-64 not available, falling back to flat repodata

The installation still works, but the client falls back to downloading/parsing the much larger flat repodata.json, which can be noticeably slower for large subdirs such as conda-forge/win-64.

From the mirror sync script, it seems that the current logic only downloads traditional metadata files:

repodata.json
repodata.json.bz2
repodata.json.zst
current_repodata.json

but does not sync:

repodata_shards.msgpack.zst

or the referenced shard files under info.shards_base_url, commonly ./shards/.

Would it be possible to add support for syncing CEP-16 sharded repodata for conda cloud channels, especially conda-forge?

Thanks.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the mirror sync script and reviewing the logic that downloads repodata.json, repodata.json.bz2, repodata.json.zst, and current_repodata.json. Trace how conda cloud channels are handled, then verify the implementation against conda-forge's repodata_shards.msgpack.zst and its referenced ./shards/ files. Done means the TUNA URLs provide the shard index and referenced shard files without the mamba/libmamba fallback warning.

Written by the indexing model from the issue text.

Assessment

Tech stack
anaconda, python
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.