conda-forge / conda-forge/conda-forge.github.io

Infrastructure update for Channel Cloning/CDN sync

Open
#2,272 17 comments 5 reactions 0 assignees View on GitHub
question
Dominant language
JavaScript
Stars
170
Forks
320
Avg merge
2d 10h
Merged PRs (30d)
5

Description

### Your question:

Hey everyone,

we (Anaconda) have been working on upgrades for the channel cloning/CDN sync infrastructure of anaconda.org, which is is used for conda-forge.

## Time planning

We are planning to switch to the new infrastructure tomorrow, 2024-08-27 at 12:30 UTC.
**No downtime is expected for users**.

Due to the amount of changes on the conda-forge channel, the initial sync after the switch to the new infrastructure will be a bit slower than the usual syncs - it is currently running in a "dry run" mode on the new infrastructure already to keep its caches up to date, and I expect the initial sync on the new infrastructure to take ~30 minutes.

We have already implemented this change for the r and anaconda channels to ensure that our maintenance process and the new infrastructure are working as desired.

## What's changing?

* The codebase has been updated to make the cloning more fault-tolerant and reliable
* The new infrastructure is more tightly integrated into our observability stack, enabling us to detect problems earlier and more granularly (e.g. a specific package that fails to clone instead of just a general cloning error)
* We're adding a `last-updated.json` file at the channel root, which contains the last update time both as UNIX and ISO8601 timestamps. We are using this directly to monitor the updates end-to-end. Check it out for the [anaconda](https://conda.anaconda.org/anaconda/last-updated.json) channel if you're interested!

## Other notes

This topic was discussed in the conda-forge meeting on 2024-08-21 and it was decided to bring it up here to raise awareness.

If there's any questions, please ask!

Contributor guide

Open the contributing guide

Research direction

The issue is an infrastructure announcement rather than a contribution task and names no files, tests, or entry points. Start by confirming whether the planned switch has already happened and locating the channel cloning/CDN sync and observability configuration. No newcomer-sized definition of done is provided.

Written by the indexing model from the issue text.

Assessment

Domain
infrastructure, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
10/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.