apache / apache/nuttx

[BUG] Network Timeout in CI Docker Workflows

Open
#14,682 0 comments 0 reactions 0 assignees View on GitHub
Arch: all Area: Build system OS: Linux Type: Bug
Dominant language
C
Stars
4k
Forks
1.7k
Avg merge
1d 17h
Merged PRs (30d)
237

Description

### Description / Steps to reproduce the issue

Something super strange about __Network Timeouts__ (pic below) in our CI Docker Workflows at GitHub Actions. Here's an example...

- First Run fails while [__downloading something from GitHub__](https://github.com/nuttxpr/nuttx/actions/runs/11535899222/job/32111488205#step:7:626)...

```text
Configuration/Tool: imxrt1050-evk/libcxxtest,CONFIG_ARM_TOOLCHAIN_GNU_EABI
curl: (28) Failed to connect to github.com port 443 after 134188 ms: Connection timed out
make[1]: *** [libcxx.defs:28: libcxx-17.0.6.src.tar.xz] Error 28
```

- Second Run fails again, while [__downloading NimBLE from GitHub__](https://github.com/nuttxpr/nuttx/actions/runs/11535899222/job/32112716849#step:7:536)...

```text
Configuration/Tool: nucleo-wb55rg/nimble,CONFIG_ARM_TOOLCHAIN_GNU_EABI
curl: (28) Failed to connect to github.com port [443](https://github.com/nuttxpr/nuttx/actions/runs/11535899222/job/32112716849#step:7:444) after 134619 ms: Connection timed out
make[2]: *** [Makefile:55: /github/workspace/sources/apps/wireless/bluetooth/nimble_context] Error 2
```

- [__Third Run succeeds.__](https://github.com/nuttxpr/nuttx/actions/runs/11535899222) Why do we keep seeing these errors: GitHub Actions with Docker, can't connect to GitHub itself?

- Is there a __Concurrent Connection Limit__ for GitHub HTTPS Connections?

We see __4 Concurrent Connections__ to GitHub HTTPS...

- [__risc-v-05__ at 00:41:06](https://github.com/nuttxpr/nuttx/actions/runs/11535899222/job/32111489166#step:7:84)

- [__xtensa-02__ at 00:41:17](https://github.com/nuttxpr/nuttx/actions/runs/11535899222/job/32111488582#step:7:510)

- [__xtensa-01__ at 00:41:34](https://github.com/nuttxpr/nuttx/actions/runs/11535899222/job/32111487874#step:7:586)

- [__risc-v-02__ at 00:41:58](https://github.com/nuttxpr/nuttx/actions/runs/11535899222/job/32111488301#step:7:532)

The __Fifth Connection__ failed: [__arm-02__ at 00:42:52](https://github.com/nuttxpr/nuttx/actions/runs/11535899222/job/32111488205#step:7:619)

- Should we use a [__Caching Proxy Server__](https://ubuntu.com/server/docs/how-to-install-a-squid-server) for curl?

```bash
$ export https_proxy=https://1.2.3.4:1234
$ curl https://github.com/...
```

- Is something misconfigured in our Docker Image? But the exact same Docker Image runs fine on [__our own Build Farm__](https://lupyuen.github.io/articles/ci2). It [__doesn't show any errors__](https://lupyuen.codeberg.page/articles/ci2.html).

- Is GitHub Actions starting our Docker Container with the wrong MTU (Network Packet Size)? 🤔

[__GitHub Actions with Smaller MTU Size__](https://github.com/actions/actions-runner-controller/issues/393)

[__Docker MTU issues and solutions__](https://mlohr.com/docker-mtu/)

- Meanwhile I'm running a script to Restart Failed Jobs on our NuttX Mirror Repo: [restart-failed-job.sh](https://github.com/lupyuen/nuttx-release/blob/main/restart-failed-job.sh)

These __Timeout Errors__ will cost us precious GitHub Minutes. The remaining jobs get killed, and restarting these killed jobs from scratch will consume extra GitHub Minutes. (The restart below costs us 6 extra GitHub Runner Hours)

1. How do we __Retry these Timeout Errors__?

1. Can we have __Restartable Builds__?

Doesn't quite make sense to kill everything and rebuild from scratch (arm6, arm7, riscv7) just because one job failed (xtensa2)

1. Or xtensa2 should __wait for others__ to finish, before it declares a timeout and croaks?

```text
Configuration/Tool: esp32s2-kaluga-1/lvgl_st7789
curl: (28) Failed to connect to github.com port 443 after 133994 ms: Connection timed out
```
[(See the __Complete Log__)](https://github.com/apache/nuttx/actions/runs/11395811301/attempts/1)

![Network Timeout at GitHub](https://lupyuen.github.io/images/ci3-timeout.png)

### On which OS does this issue occur?

[OS: Linux]

### What is the version of your OS?

Ubuntu LTS at GitHub Actions

### NuttX Version

master

### Issue Architecture

[Arch: all]

### Issue Area

[Area: Build System]

### Verification

- [X] I have verified before submitting the report.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.