Add support for syncing CEP-16 sharded repodata for conda-forge
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 495
- Forks
- 178
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I noticed that the TUNA Anaconda mirror does not seem to provide CEP-16 sharded repodata for conda-forge, while the upstream conda-forge channel already provides it.
For example, upstream conda-forge has:
curl -I https://conda.anaconda.org/conda-forge/win-64/repodata_shards.msgpack.zst
which returns:
HTTP/1.1 200 OK
Content-Type: binary/octet-stream
Content-Length: 414167
and:
curl -I https://conda.anaconda.org/conda-forge/noarch/repodata_shards.msgpack.zst
also returns:
HTTP/1.1 200 OK
Content-Type: binary/octet-stream
Content-Length: 929729
However, the TUNA mirror currently returns 404 for the same file:
curl -I https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata_shards.msgpack.zst
returns:
HTTP/1.1 404 Not Found
while the traditional flat repodata file is available:
curl -I https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/win-64/repodata.json
returns:
HTTP/1.1 200 OK
This causes recent versions of mamba / libmamba to print warnings such as:
Shard Index for conda-forge/noarch not available, falling back to flat repodata
Shard Index for conda-forge/win-64 not available, falling back to flat repodata
The installation still works, but the client falls back to downloading/parsing the much larger flat repodata.json, which can be noticeably slower for large subdirs such as conda-forge/win-64.
From the mirror sync script, it seems that the current logic only downloads traditional metadata files:
repodata.json
repodata.json.bz2
repodata.json.zst
current_repodata.json
but does not sync:
repodata_shards.msgpack.zst
or the referenced shard files under info.shards_base_url, commonly ./shards/.
Would it be possible to add support for syncing CEP-16 sharded repodata for conda cloud channels, especially conda-forge?
Thanks.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the mirror sync script and reviewing the logic that downloads repodata.json, repodata.json.bz2, repodata.json.zst, and current_repodata.json. Trace how conda cloud channels are handled, then verify the implementation against conda-forge's repodata_shards.msgpack.zst and its referenced ./shards/ files. Done means the TUNA URLs provide the shard index and referenced shard files without the mamba/libmamba fallback warning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- anaconda, python
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100