get2knowio / get2knowio/deacon
chore(ci): Podman lane intermittently fails an exec with `conmon bytes ""` — two hypotheses refuted
- Dominant language
- Rust
- Stars
- 2
- Forks
- 0
- Avg merge
- 2h 36m
- Merged PRs (30d)
- 78
Description
## Symptom
`smoke_compose_override_command::test_compose_override_command_lifecycle_runs` fails intermittently on the Podman lane:
```
Container command failed with exit code 255: Error: container create failed
(no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found ^@
```
podman is failing to start an exec session's conmon. It surfaces as a bare `deacon up failed` with `postCreateCommand … exit code 255`, which reads like a lifecycle defect and is not one. The env probe fails the same way ~0.6s earlier, so two consecutive `podman exec` calls failed.
## History
| Run | Result |
|---|---|
| #715 run 1 | different failure — a genuine test defect (hardcoded `docker`), fixed |
| #715 run 2 | `conmon` → passed on re-run |
| #717 | `conmon` |
| #720 | passed |
| #722 | `conmon` — **after** the concurrency change below |
Never reproduced locally: the test passes 5/5 serially and the full suite passes 433/433 against real rootless podman 4.9.3 in the dev container.
## Refuted: contention
`smoke-lite` was halved from 4 to 2 (#718) on the theory that concurrent rootless-podman compose projects were starving conmon — plausible, since #715 moved those projects from docker to podman for the first time. **The failure recurred unchanged at 2.** Reverted; it cost ~14% on every smoke lane and bought nothing.
Worth noting for whoever picks this up: at `max-threads = 2` the pair running concurrently was exactly the two sibling tests in that binary, one of which passed. So "two concurrent podman compose projects" is not sufficient to cause it either.
## Refuted: container not running
`get_primary_container_id` (`crates/core/src/compose.rs:1723`) matches a service **by name with no state check**, while its sibling `handle_port_events` filters on `s.state == "running"`. A not-yet-running or already-exited container looked like a strong candidate.
**Measured** against rootless podman 4.9.3:
- `podman compose up -d` returns with the container already `running`; polling `compose ps` immediately never showed an intermediate state.
- exec into an **exited** compose container gives `can only create exec sessions on running containers: container state improper` — a clean, *different* error.
So this is not it. (The missing state check is still an asymmetry worth tidying, but it is not this bug and should not be "fixed" as though it were.)
## What is known
- It is podman failing to *create* the exec session, not deacon misusing it.
- `readObjectStart: expect { or n, but found ^@` means podman read NUL from conmon's sync pipe — conmon died or wrote nothing.
- Typical causes for that shape are resource ceilings (pids, fds, memory / OOM-kill of conmon), not container state.
- No OOM or resource message appears in the job log, but nothing currently *looks*.
## Next step is evidence, not a third hypothesis
A failure-only diagnostics step is added to the Podman job: memory, load, `ulimit -a`, `pids.max`, conmon presence and version, `podman info`, container states, and `dmesg` tail. Everything is `|| true` and `if: failure()`, so it reports and never masks the real failure.
- [ ] Wait for the next occurrence and read the diagnostics.
- [ ] If it is a resource ceiling, raise it or reduce whole-suite concurrency — not `smoke-lite` alone, which has been tried.
- [ ] Only then consider a change to deacon.
Deliberately **not** doing: quarantining the test, adding a retry, or reducing concurrency further. Each would hide the signal, and two of them would hide it permanently.
Contributor guide
Research direction
Wait for the next Podman-lane failure and inspect the failure-only diagnostics for resource ceilings, conmon, podman info, container states, and the dmesg tail. Use the smoke_compose_override_command::test_compose_override_command_lifecycle_runs failure as the trigger; if resources are implicated, adjust the ceiling or whole-suite concurrency, otherwise defer a deacon change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, rust
- Domain
- ci-cd, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100