bazelbuild / bazelbuild/bazelisk

bazelisk crashes within enroot container running on slurm cluster

Open
#665 5 comments 0 reactions 0 assignees View on GitHub
P4
Dominant language
Go
Stars
2.7k
Forks
424
PR merge metrics
No merged PRs in 30d

Description

I am running bazelisk as part of my GitHub CI which in turn uses an enroot container as part of a slurm job. Bazelisk is crashing on me immediately after launch. Here is a minimum command to reproduce it and the corresponding error message:

```bash
wget https://github.com/bazelbuild/bazelisk/releases/download/v1.25.0/bazelisk-linux-amd64
chmod +x bazelisk-linux-amd64
./bazelisk-linux-amd64

2025/03/03 05:23:55 Downloading https://releases.bazel.build/8.1.1/release/bazel-8.1.1-linux-x86_64...
Downloading: 58 MB out of 58 MB (100%)
WARNING: Invoking Bazel in batch mode since it is not invoked from within a workspace (below a directory having a MODULE.bazel file).
Extracting Bazel installation...
WARNING: Ignoring JAVA_HOME, because it must point to a JDK, not a JRE.
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.ExceptionInInitializerError
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.(SkyframeExecutor.java:449)
at com.google.devtools.build.lib.skyframe.BazelSkyframeExecutorConstants.newBazelSkyframeExecutorBuilder(BazelSkyframeExecutorConstants.java:45)
at com.google.devtools.build.lib.skyframe.SequencedSkyframeExecutorFactory.create(SequencedSkyframeExecutorFactory.java:44)
at com.google.devtools.build.lib.runtime.WorkspaceBuilder.build(WorkspaceBuilder.java:118)
at com.google.devtools.build.lib.runtime.BlazeRuntime.initWorkspace(BlazeRuntime.java:274)
at com.google.devtools.build.lib.runtime.BlazeRuntime.newRuntime(BlazeRuntime.java:1440)
at com.google.devtools.build.lib.runtime.BlazeRuntime.batchMain(BlazeRuntime.java:1031)
at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:866)
at com.google.devtools.build.lib.bazel.Bazel.main(Bazel.java:104)
Caused by: java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Unknown Source)
at java.base/sun.nio.fs.UnixFileSystem.getPath(Unknown Source)
at java.base/java.nio.file.Path.of(Unknown Source)
at java.base/java.nio.file.Paths.get(Unknown Source)
at java.base/jdk.internal.platform.CgroupUtil.lambda$readStringValue$1(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/jdk.internal.platform.CgroupUtil.readStringValue(Unknown Source)
at java.base/jdk.internal.platform.CgroupSubsystemController.getStringValue(Unknown Source)
at java.base/jdk.internal.platform.CgroupSubsystemController.getLongValue(Unknown Source)
at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getLongValue(Unknown Source)
at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getHierarchical(Unknown Source)
at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.initSubSystem(Unknown Source)
at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getInstance(Unknown Source)
at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(Unknown Source)
at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(Unknown Source)
at java.base/jdk.internal.platform.CgroupMetrics.getInstance(Unknown Source)
at java.base/jdk.internal.platform.SystemMetrics.instance(Unknown Source)
at java.base/jdk.internal.platform.Metrics.systemMetrics(Unknown Source)
at java.base/jdk.internal.platform.Container.metrics(Unknown Source)
at jdk.management/com.sun.management.internal.OperatingSystemImpl.(Unknown Source)
at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean(Unknown Source)
at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap(Unknown Source)
at java.management/sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans(Unknown Source)
at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(Unknown Source)
at java.management/java.lang.management.ManagementFactory.getOperatingSystemMXBean(Unknown Source)
at com.google.devtools.build.lib.util.ResourceUsage.(ResourceUsage.java:46)
... 9 more
```

I can't figure out what the problem with the environment is. Any help would be appreciated.

EDIT: I used something like this to launch the slurm job.

```bash
srun \
--job-name=debug_ci_runner \
--time=1:00:00 \
--cpus-per-task=16 \
--gres=gpu:1 \
--container-image=/path/to/my/ci/images/gh_runner.sqsh \
--container-remap-root \
--no-container-mount-home \
--container-mounts=/usr/share/sunk:/usr/share/sunk \
--container-workdir=/actions-runner/ \
--pty /bin/bash
```

Contributor guide

Open the contributing guide

Research direction

Start with the bazelisk-linux-amd64 reproducer and the shown srun container command, then trace the stack from Bazel's ResourceUsage initialization into Java's cgroup handling. Compare the behavior inside and outside the enroot/Slurm environment; done means identifying the environmental trigger and confirming a reliable fix or documented workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, go, java
Domain
build-system, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.