containerd / containerd/nerdctl

[CI]: State of the CI

Open
#3,924 5 comments 0 reactions 0 assignees View on GitHub
area/ci
Dominant language
Go
Stars
10.4k
Forks
826
Avg merge
1d 23h
Merged PRs (30d)
44

Description

### Description

In the past week(s) or so, the CI has been increasingly unstable.

While the root cause is unclear, I suspect github networking is somewhat under pressure, or maybe we are in a tier that is somehow throttled?

Specifically:
- ubuntu server are timeouting a lot (unable to connect): https://github.com/containerd/nerdctl/actions/runs/13511118022/job/37751406679?pr=3920#step:7:2624 (never happened before)
- golangci fails to reach its server to retrieve their schema: https://github.com/containerd/nerdctl/actions/runs/13476112452/job/37655514638?pr=3890#step:5:38 (never happened before)
- github cache retrieval has been increasingly slugish: instead of mere seconds to retrieve the manifest, it can now take up to 10 minutes
- docker hub increasingly times-out or give us the dreaded 429 (somewhat orthogonal to the rest)

Maybe these are all related (eg: github networking degraded) - or maybe they are not, and it is a coincidence.

One way or the other, we are getting to the point where it is hard to get a green build on first try (on top of our test flakyness, which is a long fought battle on its own).

While some easy/localized actions can be taken to reduce outbound traffic for routine operations (#3915), I only see two possibilities moving forward:
a. we get some help / information from github about network quality
b. we take a serious hard look at how we do things and significantly reduce our outbound dependencies

I do not have insider contact for a. Does anyone have some?

For b.:
- we could consider getting rid entirely of reliance on Docker Hub and host everything we need on ghcr instead (on the assumption it will be better), and systematically hunt down and remove unneeded outbound traffic (eg: golangci for eg)
- we rethink both the way we build and use our base image and the way we use github cache, which has been a growing PITA (quota, slugishness)

Do people have thoughts about all this?

### Steps to reproduce the issue

1.
2.
3.

### Describe the results you received and expected

na

### What version of nerdctl are you using?

main

### Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

### Host information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.