bazelbuild / bazelbuild/bazel

(Why) do we need test-setup.sh?

Open
#18,980 5 comments 2 reactions 0 assignees View on GitHub
not stale P3 team-Remote-Exec type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the feature request:

All tests are currently launched through a shell script named `test-setup.sh`. Even though that works, I've observed that it is somewhat of an impediment if you're dealing with slightly non-standard use cases.

For example, `test-setup.sh` has a `#!/usr/bin/env bash` at the top, meaning that you can only run tests on platforms that support/ship with Bash.

Furthermore, when replaying a previously failed test, it's a bit annoying to get tests to run under tracers/debuggers, as they all need to 'jump over' the shell script, so to speak.

My question is, why do we need `test-setup.sh` in the first place? Are there certain things that it does that could also just be solved by calling into SpawnRunner/the remote execution system differently? For example, this code is only setting an environment variable, which we can also do trivially through other means:
```
# Declare that the executable is running in a `bazel test` environment
# This allows test frameworks to enable output to the unprefixed environment variable
# For example, if `BAZEL_TEST` and `XML_OUTPUT_FILE` are defined, write JUnit output
export BAZEL_TEST=1
```
Or are there genuine features that are missing? If so, could we maybe work with the remote execution working group to get those features added to the protocol?

One example I can think of is this:

```
# Bazel sets some environment vars to relative paths to improve caching and
# support remote execution, where the absolute path may not be known to Bazel.
# Convert them to absolute paths here before running the actual test.
is_absolute "$TEST_PREMATURE_EXIT_FILE" ||
TEST_PREMATURE_EXIT_FILE="$PWD/$TEST_PREMATURE_EXIT_FILE"
```

Maybe we could extend REv2's `Command` message to have a dedicated field/feature for this?

### What underlying problem are you trying to solve with this feature?

Increase portability across different remote execution platforms. Make it easier to being able to debug tests.

### Which operating system are you running Bazel on?

n/a

### What is the output of `bazel info release`?

n/a

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

n/a

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

```text
n/a
```

### Have you found anything relevant by searching the web?

I see that someone filed an issue a couple of hours ago to add automatic injection of timestamps to test logs: #18973
But this information is also provided as part of REv2's `ExecutedActionMetadata`. Bazel could just display that information if needed.

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading test-setup.sh and tracing how SpawnRunner or the remote execution system invokes it. Compare its environment-variable and path-normalization behavior with the REv2 Command and ExecutedActionMetadata mentioned in the issue; done means documenting which responsibilities can be removed, relocated, or require protocol changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash
Domain
build-system, distributed-systems, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.