test_tower_skip_rss_arm: over half the suite fails, fake health server exits before ready
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: -
python3 -m unittest tests.scripts.test_tower_skip_rss_arm -v fails on a
clean, unmodified main (bcade48d6) on this box: 12 failures + 1 error out
of 24 tests, all with the same shape --
AssertionError: 5 != 0 : LEG default FAILED rc=5
arm: the server exited before it was ready; see /tmp/tower-skip-arm-XXXX/out/default.log
arm default: the server never became ready
This suite drives the harness's real run_arm/readiness-poll/teardown shell
functions against a fake Python /health server the test writes to a temp
file (FAKE_SERVER in the test module) -- it does not touch the GPU, ROCm, or
build-hip/examples/vllm-cli, which is present and unrelated. No docker
container, GPU lock, or stray server process was left over from other work in
this session at the time of the failure (checked docker ps -a, fuser $HOME/gpu.lock, pgrep) -- all clean. Root cause not yet identified; the
warmup.log files under /tmp/tower-skip-arm-* are cleaned up by the test's
TemporaryDirectory teardown before they can be inspected.
Filed without a fix in the same flow: found as a side effect of unrelated
work (ORACLE-VLLM-ROCM-GFX1200-DOCKER, #2961), and root-causing why a fake
Python health server dies before it starts needs its own investigation
(capture the log before teardown, check for a port or sandbox restriction
specific to this box) rather than a guess landed in a hurry.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run python3 -m unittest tests.scripts.test_tower_skip_rss_arm -v and inspect tests.scripts.test_tower_skip_rss_arm, especially FAKE_SERVER and the run_arm readiness and teardown functions. Preserve warmup.log or default.log before TemporaryDirectory cleanup, then investigate why the fake Python /health server exits and confirm the full 24-test suite passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100