containerd / containerd/nerdctl

[LOGS]: logs are unreliable and randomly miss data

Open
#4,243 5 comments 0 reactions 0 assignees View on GitHub
area/logging bug
Dominant language
Go
Stars
10.4k
Forks
826
Avg merge
1d 23h
Merged PRs (30d)
44

Description

### Description

Repeatedly running tests will eventually bring failures.

In this case copied below, a simple `run echo` does successfully print on stdout, but there is nothing in the log file.

```
--- FAIL: TestLogsAfterRestartingContainer (6.44s)
container_logs_test.go:365:

+============================================================================================================+
| 🧽 | "TestLogsAfterRestartingContainer": initial cleanup |
+============================================================================================================+

container_logs_test.go:338:

+------------------------------------------------------------------------------------------------------------+
| ➡️ | ⚙️ /usr/local/bin/nerdctl rm -f testlogsafterrestartingcontainer-a8f1fb79 |
+------------------------------------------------------------------------------------------------------------+
| | 🟠 time="2025-05-12T22:04:21-07:00" level=error msg="1 errors:\nno such container: testlogsafter |
| | restartingcontainer-a8f1fb79" |
+------------------------------------------------------------------------------------------------------------+
| 🌱 | XDG_SESSION_TYPE=tty |
| | HOME=/home/dmp.linux |
| | XDG_SESSION_CLASS=user |
| | XDG_SESSION_ID=2 |
| | XDG_RUNTIME_DIR=/run/user/501 |
| | PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/sbin:/sbin: |
| | /usr/sbin:/sbin:/usr/local/go/bin:/home/dmp.linux/go/bin |
| | NERDCTL_TOML=/tmp/TestLogsAfterRestartingContainer4032231042/003/nerdctl.toml |
| | DOCKER_CONFIG=/tmp/TestLogsAfterRestartingContainer4032231042/003 |
+------------------------------------------------------------------------------------------------------------+
| ⏰ | <1s (limit: 3m0s) |
+------------------------------------------------------------------------------------------------------------+
| 📁 | /tmp/TestLogsAfterRestartingContainer4032231042/002 |
+------------------------------------------------------------------------------------------------------------+

container_logs_test.go:365:

+============================================================================================================+
| 🏗 | "TestLogsAfterRestartingContainer": setup |
+============================================================================================================+

container_logs_test.go:332:

+------------------------------------------------------------------------------------------------------------+
| ➡️ | ⚙️ /usr/local/bin/nerdctl run --name testlogsafterrestartingcontainer-a8f1fb79 ghcr.io/stargz-co |
| | ntainers/alpine:3.13-org printf 'Hello World! |
| | There is no newline' |
+------------------------------------------------------------------------------------------------------------+
| | 🟢 'Hello World! |
| | There is no newline' |
+------------------------------------------------------------------------------------------------------------+
| 🌱 | XDG_SESSION_TYPE=tty |
| | HOME=/home/dmp.linux |
| | XDG_SESSION_CLASS=user |
| | XDG_SESSION_ID=2 |
| | XDG_RUNTIME_DIR=/run/user/501 |
| | PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/sbin:/sbin: |
| | /usr/sbin:/sbin:/usr/local/go/bin:/home/dmp.linux/go/bin |
| | DOCKER_CONFIG=/tmp/TestLogsAfterRestartingContainer4032231042/003 |
| | NERDCTL_TOML=/tmp/TestLogsAfterRestartingContainer4032231042/003/nerdctl.toml |
+------------------------------------------------------------------------------------------------------------+
| ⏰ | <1s (limit: 3m0s) |
+------------------------------------------------------------------------------------------------------------+
| 📁 | /tmp/TestLogsAfterRestartingContainer4032231042/002 |
+------------------------------------------------------------------------------------------------------------+

container_logs_test.go:365:
⤵️️ "TestLogsAfterRestartingContainer": into subtests prep
--- FAIL: TestLogsAfterRestartingContainer/logs_-f_works (0.09s)
container_logs_test.go:365:

+============================================================================================================+
| 🚀 | "TestLogsAfterRestartingContainer/logs_-f_works": starting test! |
+============================================================================================================+
| ⏳ | /tmp/TestLogsAfterRestartingContainerlogs_-f_works3636253693/001/471579cc |
+============================================================================================================+
| | |
+============================================================================================================+
| config | { |
| | "Namespace": "nerdctl-test", |
| | "PrivateMode": "" |
| | } |
+============================================================================================================+
| labels | {} |
+============================================================================================================+

container_logs_test.go:365:

+------------------------------------------------------------------------------------------------------------+
| ➡️ | ⚙️ /usr/local/bin/nerdctl logs -f testlogsafterrestartingcontainer-a8f1fb79 |
+------------------------------------------------------------------------------------------------------------+
| 🌱 | XDG_SESSION_TYPE=tty |
| | HOME=/home/dmp.linux |
| | XDG_SESSION_CLASS=user |
| | XDG_SESSION_ID=2 |
| | XDG_RUNTIME_DIR=/run/user/501 |
| | PATH=/usr/local/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/sbin:/sbin: |
| | /usr/sbin:/sbin:/usr/local/go/bin:/home/dmp.linux/go/bin |
| | DOCKER_CONFIG=/tmp/TestLogsAfterRestartingContainerlogs_-f_works3636253693/003 |
| | NERDCTL_TOML=/tmp/TestLogsAfterRestartingContainerlogs_-f_works3636253693/003/nerdctl.toml |
+------------------------------------------------------------------------------------------------------------+
| ⏰ | <1s (limit: 3m0s) |
+------------------------------------------------------------------------------------------------------------+
| 📁 | /tmp/TestLogsAfterRestartingContainerlogs_-f_works3636253693/002 |
+------------------------------------------------------------------------------------------------------------+

container_logs_test.go:365:

<<<<<<<<<<<<<<<<<<<<
🖊️ Inspecting output (contains)
👀 testing: ``
❌ FAILED! ~= `Hello World!`
>>>>>>>>>>>>>>>>>>>>

container_logs_test.go:365:

<<<<<<<<<<<<<<<<<<<<
🖊️ Inspecting output (contains)
👀 testing: ``
❌ FAILED! ~= `There is no newline`
>>>>>>>>>>>>>>>>>>>>

```

### Steps to reproduce the issue

Hammer tests.

### Describe the results you received and expected

Not loose data.

### What version of nerdctl are you using?

main

### Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

### Host information

Logs codebase is intricate, and even with some moderate nerdctl experience, it is not trivial to even grok what is going on.

Are we even responsible for writing logs at any point? Or is it containerd only?

Is there a reason to believe that containerd could delay writing logs after a container has exited?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.