apple / apple/container

[Bug]: `make integration` can't be run repeatedly, integration tests can't be executed concurrently

Open
#686 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
49.9k
Forks
1.8k
Avg merge
1d 20h
Merged PRs (30d)
22

Description

### I have done the following

- [x] I have searched the existing issues
- [x] If possible, I've reproduced the issue using the 'main' branch of this project

### Steps to reproduce

1. Run `make cleancontent`.
2. Run `make integration` two times.

Alternatively:
1. Run `make cleancontent`.
2. Revert https://github.com/apple/container/pull/200 to enable a parallel execution of integration tests.
3. Run `make integration` once.

### Current behavior

```
✘ Test testNetworkCreateAndUse() recorded an issue at TestCLINetwork.swift:85:13: Expectation failed: success
↳ Request to http://192.168.65.2:59242 failed after 10 retries
✘ Test testNetworkCreateAndUse() failed after 62.447 seconds with 1 issue.
✘ Suite TestCLINetwork failed after 62.448 seconds with 1 issue.
✘ Test run with 3 tests in 1 suite failed after 62.448 seconds with 1 issue.
```

```
✘ Test testForwardTCP() recorded an issue at TestCLIRunOptions.swift:512:13: Expectation failed: success
↳ Request to http://127.0.0.1:50477 failed after 10 retries
✘ Test testForwardTCP() failed after 75.191 seconds with 1 issue.
✘ Suite TestCLIRunCommand failed after 75.191 seconds with 1 issue.
✘ Test run with 20 tests in 1 suite failed after 75.192 seconds with 1 issue.
```

```
✘ Test testBuildNetworkAccess() recorded an issue at CLIBuilderTest.swift:168:10: Caught error: executionFailed("build failed: stdout= stderr=\u{1B}[33mWarning!\u{1B}[0m Running debug build. Performance may be degraded.\n#1 [resolver] fetching image...ghcr.io/linuxcontainers/alpine:3.20\n#1 DONE 0.0s\n\n#2 [internal] load .dockerignore\n#2 transferring context: 2B done\n#2 DONE 0.0s\n\n#3 oci-layout://ghcr.io/linuxcontainers/alpine:3.20@sha256:28a33e0390da2b341d05e53803c6766977039baedfc8a82a4910d5acc4755b66\n#3 resolve ghcr.io/linuxcontainers/alpine:3.20@sha256:28a33e0390da2b341d05e53803c6766977039baedfc8a82a4910d5acc4755b66 0.1s done\n#3 sha256:4cbb6abcfc2406820012ef89d697859152a729e7dbde6a4217f99136708f716d 7.55kB / 7.55kB done\n#3 sha256:94e9d8af22013aabf0edcaf42950c88b0a1350c3a9ce076d61b98a535a673dd9 4.09MB / 4.09MB 0.0s done\n#3 extracting sha256:94e9d8af22013aabf0edcaf42950c88b0a1350c3a9ce076d61b98a535a673dd9 0.1s done\n#3 DONE 0.2s\n\n#3 oci-layout://ghcr.io/linuxcontainers/alpine:3.20@sha256:28a33e0390da2b341d05e53803c6766977039baedfc8a82a4910d5acc4755b66\n#3 extracting sha256:4cbb6abcfc2406820012ef89d697859152a729e7dbde6a4217f99136708f716d done\n#3 DONE 0.2s\n\n#4 [linux/arm64 1/2] RUN nc -zv 8.8.8.8 53 || exit 1\n#4 3.135 nc: 8.8.8.8 (8.8.8.8:53): Host is unreachable\n#4 ERROR: process \"/bin/sh -c nc -zv ${ADDRESS%:*} ${ADDRESS##*:} || exit 1\" did not complete successfully: exit code: 1\n------\n > [linux/arm64 1/2] RUN nc -zv 8.8.8.8 53 || exit 1:\n3.135 nc: 8.8.8.8 (8.8.8.8:53): Host is unreachable\n------\nError: unknown (2): failed to solve: process \"/bin/sh -c nc -zv ${ADDRESS%:*} ${ADDRESS##*:} || exit 1\" did not complete successfully: exit code: 1\n")
✘ Test testBuildNetworkAccess() failed after 4.978 seconds with 1 issue.
```

```
✘ Suite CLIBuilderTest failed after 48.434 seconds with 1 issue.
✘ Suite TestCLIBuildBase failed after 61.151 seconds with 1 issue.
↳ /* CLIBuildBase is the base class used for creating builder tests. Subtests classes
// for these tests are nested in extensions of CLIBuildBase so that we can set
// the serialized parallelization attribute across all builder tests.
*/
✘ Test run with 21 tests in 5 suites failed after 61.151 seconds with 1 issue.
```

### Expected behavior

All tests pass.

### Environment

```markdown
N/A
```

### Relevant log output

```shell
N/A
```

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.