get2knowio / get2knowio/deacon

chore(ci): Podman lane intermittently fails an exec with `conmon bytes ""` — two hypotheses refuted

Open
#723 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2
Forks
0
Avg merge
2h 36m
Merged PRs (30d)
78

Description

## Symptom

`smoke_compose_override_command::test_compose_override_command_lifecycle_runs` fails intermittently on the Podman lane:

```
Container command failed with exit code 255: Error: container create failed
(no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found ^@
```

podman is failing to start an exec session's conmon. It surfaces as a bare `deacon up failed` with `postCreateCommand … exit code 255`, which reads like a lifecycle defect and is not one. The env probe fails the same way ~0.6s earlier, so two consecutive `podman exec` calls failed.

## History

| Run | Result |
|---|---|
| #715 run 1 | different failure — a genuine test defect (hardcoded `docker`), fixed |
| #715 run 2 | `conmon` → passed on re-run |
| #717 | `conmon` |
| #720 | passed |
| #722 | `conmon` — **after** the concurrency change below |

Never reproduced locally: the test passes 5/5 serially and the full suite passes 433/433 against real rootless podman 4.9.3 in the dev container.

## Refuted: contention

`smoke-lite` was halved from 4 to 2 (#718) on the theory that concurrent rootless-podman compose projects were starving conmon — plausible, since #715 moved those projects from docker to podman for the first time. **The failure recurred unchanged at 2.** Reverted; it cost ~14% on every smoke lane and bought nothing.

Worth noting for whoever picks this up: at `max-threads = 2` the pair running concurrently was exactly the two sibling tests in that binary, one of which passed. So "two concurrent podman compose projects" is not sufficient to cause it either.

## Refuted: container not running

`get_primary_container_id` (`crates/core/src/compose.rs:1723`) matches a service **by name with no state check**, while its sibling `handle_port_events` filters on `s.state == "running"`. A not-yet-running or already-exited container looked like a strong candidate.

**Measured** against rootless podman 4.9.3:

- `podman compose up -d` returns with the container already `running`; polling `compose ps` immediately never showed an intermediate state.
- exec into an **exited** compose container gives `can only create exec sessions on running containers: container state improper` — a clean, *different* error.

So this is not it. (The missing state check is still an asymmetry worth tidying, but it is not this bug and should not be "fixed" as though it were.)

## What is known

- It is podman failing to *create* the exec session, not deacon misusing it.
- `readObjectStart: expect { or n, but found ^@` means podman read NUL from conmon's sync pipe — conmon died or wrote nothing.
- Typical causes for that shape are resource ceilings (pids, fds, memory / OOM-kill of conmon), not container state.
- No OOM or resource message appears in the job log, but nothing currently *looks*.

## Next step is evidence, not a third hypothesis

A failure-only diagnostics step is added to the Podman job: memory, load, `ulimit -a`, `pids.max`, conmon presence and version, `podman info`, container states, and `dmesg` tail. Everything is `|| true` and `if: failure()`, so it reports and never masks the real failure.

- [ ] Wait for the next occurrence and read the diagnostics.
- [ ] If it is a resource ceiling, raise it or reduce whole-suite concurrency — not `smoke-lite` alone, which has been tried.
- [ ] Only then consider a change to deacon.

Deliberately **not** doing: quarantining the test, adding a retry, or reducing concurrency further. Each would hide the signal, and two of them would hide it permanently.

Contributor guide

Open the contributing guide

Research direction

Wait for the next Podman-lane failure and inspect the failure-only diagnostics for resource ceilings, conmon, podman info, container states, and the dmesg tail. Use the smoke_compose_override_command::test_compose_override_command_lifecycle_runs failure as the trigger; if resources are implicated, adjust the ceiling or whole-suite concurrency, otherwise defer a deacon change.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust
Domain
ci-cd, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.