apache / apache/datafusion-ballista

CI: TPC-H SF10 failure artifact is missing executor.log

Open
#2,147 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Describe the bug**

When a `TPC-H SF10` job fails, the `Upload cluster logs on failure` step in `.github/workflows/tpch.yml` is supposed to attach both `scheduler.log` and `executor.log`. Only `scheduler.log` ever makes it into the artifact.

That is the more valuable of the two to lose. Most SF10 failures are task-level (spill, shuffle write, OOM, disk), and those only show up in the executor log. Without it you are left with the 200 line tail the cleanup trap echoes into the job output, which is usually not enough to see what led up to the failure.

**To Reproduce**

Two independent failed runs, different failure modes, same result:

- Run [29923718090](https://github.com/apache/datafusion-ballista/actions/runs/29923718090) (`No space left on device` during query 21), artifact `tpch-sf10-cluster-logs-aqe-on-mpt` contains only `__w/_temp/scheduler.log`.
- Run [29749959848](https://github.com/apache/datafusion-ballista/actions/runs/29749959848) (executor registration failure), artifact `tpch-sf10-cluster-logs` contains only `__w/_temp/scheduler.log`.

```
$ gh run download 29923718090 -n tpch-sf10-cluster-logs-aqe-on-mpt -D art
$ find art -type f
art/__w/_temp/scheduler.log
```

In both cases `executor.log` demonstrably existed at upload time, because the cleanup trap in the same step successfully ran `tail -n 200 "$EXECUTOR_LOG"` and its output is visible in the job log a fraction of a second earlier.

**Expected behavior**

The failure artifact contains both `scheduler.log` and `executor.log`.

**Additional context**

The step config is:

```yaml
- name: Upload cluster logs on failure
if: failure()
uses: actions/upload-artifact@v7
with:
name: tpch-sf10-cluster-logs-${{ matrix.slug }}
path: |
${{ runner.temp }}/scheduler.log
${{ runner.temp }}/executor.log
if-no-files-found: ignore
```

I have not pinned down the mechanism, so treat the following as clues rather than a diagnosis. From the upload step's own output:

```
path: /home/runner/work/_temp/scheduler.log
/home/runner/work/_temp/executor.log

Multiple search paths detected. Calculating the least common ancestor of all paths
The least common ancestor is /. This will be the root directory of the artifact
With the provided path, there will be 1 file uploaded
```

Two things stand out. The action only resolved one of the two search paths, and the least common ancestor came out as `/` rather than the shared `/home/runner/work/_temp` directory, which suggests the two paths did not resolve to the same root. Note also that this is a container job (`image: amd64/rust`), `${{ runner.temp }}` expands to the host path `/home/runner/work/_temp`, and the file lands in the artifact under the container path `/__w/_temp`. Some interaction between the container path mapping and multi-path glob resolution looks like the likely culprit, but that needs confirming.

Since `if-no-files-found: ignore` is set, the miss is silent.

Possible fixes, roughly in order of how much I like them:

- Point `path` at the directory or a glob (`${{ runner.temp }}/*.log`) so only a single search path is involved and the LCA logic is not exercised.
- Write both logs into a dedicated subdirectory (e.g. `${{ runner.temp }}/cluster-logs/`) and upload that one directory.
- Switch `if-no-files-found` to `warn` so a future silent miss is at least visible in the job log.

Found while investigating the SF10 disk exhaustion in #1996.

Contributor guide

Open the contributing guide

Research direction

Start with .github/workflows/tpch.yml and the Upload cluster logs on failure step. Reproduce or inspect the referenced runs with gh run download, then trace how the container path mapping and multiple upload paths resolve. Done means a failed SF10 artifact contains both scheduler.log and executor.log, with misses no longer silent.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions
Domain
ci-cd
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.