(Why) do we need test-setup.sh?
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the feature request:
All tests are currently launched through a shell script named `test-setup.sh`. Even though that works, I've observed that it is somewhat of an impediment if you're dealing with slightly non-standard use cases.
For example, `test-setup.sh` has a `#!/usr/bin/env bash` at the top, meaning that you can only run tests on platforms that support/ship with Bash.
Furthermore, when replaying a previously failed test, it's a bit annoying to get tests to run under tracers/debuggers, as they all need to 'jump over' the shell script, so to speak.
My question is, why do we need `test-setup.sh` in the first place? Are there certain things that it does that could also just be solved by calling into SpawnRunner/the remote execution system differently? For example, this code is only setting an environment variable, which we can also do trivially through other means:
```
# Declare that the executable is running in a `bazel test` environment
# This allows test frameworks to enable output to the unprefixed environment variable
# For example, if `BAZEL_TEST` and `XML_OUTPUT_FILE` are defined, write JUnit output
export BAZEL_TEST=1
```
Or are there genuine features that are missing? If so, could we maybe work with the remote execution working group to get those features added to the protocol?
One example I can think of is this:
```
# Bazel sets some environment vars to relative paths to improve caching and
# support remote execution, where the absolute path may not be known to Bazel.
# Convert them to absolute paths here before running the actual test.
is_absolute "$TEST_PREMATURE_EXIT_FILE" ||
TEST_PREMATURE_EXIT_FILE="$PWD/$TEST_PREMATURE_EXIT_FILE"
```
Maybe we could extend REv2's `Command` message to have a dedicated field/feature for this?
### What underlying problem are you trying to solve with this feature?
Increase portability across different remote execution platforms. Make it easier to being able to debug tests.
### Which operating system are you running Bazel on?
n/a
### What is the output of `bazel info release`?
n/a
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
n/a
### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?
```text
n/a
```
### Have you found anything relevant by searching the web?
I see that someone filed an issue a couple of hours ago to add automatic injection of timestamps to test logs: #18973
But this information is also provided as part of REv2's `ExecutedActionMetadata`. Bazel could just display that information if needed.
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start by reading test-setup.sh and tracing how SpawnRunner or the remote execution system invokes it. Compare its environment-variable and path-normalization behavior with the REv2 Command and ExecutedActionMetadata mentioned in the issue; done means documenting which responsibilities can be removed, relocated, or require protocol changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash
- Domain
- build-system, distributed-systems, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100