task failure due to reboot request time out should print off all available logs up to that point
- Dominant language
- Go
- Stars
- 127
- Forks
- 81
- PR merge metrics
- No merged PRs in 30d
Description
Currently when debugging tests where the system is rebooted and it fails to come back after a reboot, we receive no information about the text execution like we would if the `execute` section had failed, etc. See for example this failure:
```
sergio@cachiomachine:~/workspace/snapcore/snapd$ spread external:ubuntu-core-20-64:tests/core/uc20-recovery
2020-10-02 16:04:32 Project content is packed for delivery (4.66MB).
2020-10-02 16:04:32 If killed, discard servers with: spread -reuse-pid=233681 -discard
2020-10-02 16:04:32 Allocating external:ubuntu-core-20-64...
2020-10-02 16:04:32 Waiting for external:ubuntu-core-20-64 to make SSH available at localhost:8022...
2020-10-02 16:04:32 Allocated external:ubuntu-core-20-64.
2020-10-02 16:04:32 Connecting to external:ubuntu-core-20-64...
2020-10-02 16:04:32 Connected to external:ubuntu-core-20-64 at localhost:8022.
2020-10-02 16:04:32 Sending project content to external:ubuntu-core-20-64...
2020-10-02 16:04:33 Preparing external:ubuntu-core-20-64 (external:ubuntu-core-20-64)...
2020-10-02 16:04:33 Preparing external:ubuntu-core-20-64:tests/core/ (external:ubuntu-core-20-64)...
2020-10-02 16:07:20 Preparing external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64)...
2020-10-02 16:07:22 Executing external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) (1/1)...
2020-10-02 16:08:56 Rebooting on external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) as requested...
2020-10-02 16:13:12 Rebooting on external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) as requested...
2020-10-02 16:18:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Current output:
-----
mkdir: cannot create directory ‘/home/gopath’: File exists
sudo: unable to resolve host ubuntu: Temporary failure in name resolution
-----
.
2020-10-02 16:23:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:28:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:33:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:38:12 WARNING: external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) running late. Output unchanged.
2020-10-02 16:43:11 Error executing external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) : kill-timeout reached after external:ubuntu-core-20-64 (external:ubuntu-core-20-64:tests/core/uc20-recovery) reboot request
2020-10-02 16:43:11 Error debugging external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) : ssh: unexpected packet in response to channel open:
2020-10-02 16:43:11 Restoring external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64)...
2020-10-02 16:43:11 Error restoring external:ubuntu-core-20-64:tests/core/uc20-recovery (external:ubuntu-core-20-64) : EOF
2020-10-02 16:43:11 Restoring external:ubuntu-core-20-64:tests/core/ (external:ubuntu-core-20-64)...
2020-10-02 16:43:11 Error restoring external:ubuntu-core-20-64:tests/core/ (external:ubuntu-core-20-64) : EOF
2020-10-02 16:43:11 Restoring external:ubuntu-core-20-64 (external:ubuntu-core-20-64)...
2020-10-02 16:43:11 Error restoring external:ubuntu-core-20-64 (external:ubuntu-core-20-64) : EOF
2020-10-02 16:43:11 Discarding external:ubuntu-core-20-64...
2020-10-02 16:43:11 Successful tasks: 0
2020-10-02 16:43:11 Aborted tasks: 0
2020-10-02 16:43:11 Failed tasks: 1
- external:ubuntu-core-20-64:tests/core/uc20-recovery
2020-10-02 16:43:11 Failed task restore: 1
- external:ubuntu-core-20-64:tests/core/uc20-recovery
2020-10-02 16:43:11 Failed suite restore: 1
- external:ubuntu-core-20-64:tests/core/
2020-10-02 16:43:11 Failed project restore: 1
- external:ubuntu-core-20-64:project
error: unsuccessful run
```
The only information we get is the "is taking too long Current Output" message which is not enough information, and additionally the debug section is not executed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how reboot requests reach the kill-timeout path and compare it with the existing handling for failures in the execute section. Ensure the failure reports all output collected before the timeout and still runs the relevant debugging step; the example output in the issue provides the expected scenario to reproduce and verify.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100