bazelbuild / bazelbuild/bazelisk
bazelisk crashes within enroot container running on slurm cluster
- Dominant language
- Go
- Stars
- 2.7k
- Forks
- 424
- PR merge metrics
- No merged PRs in 30d
Description
I am running bazelisk as part of my GitHub CI which in turn uses an enroot container as part of a slurm job. Bazelisk is crashing on me immediately after launch. Here is a minimum command to reproduce it and the corresponding error message:
```bash
wget https://github.com/bazelbuild/bazelisk/releases/download/v1.25.0/bazelisk-linux-amd64
chmod +x bazelisk-linux-amd64
./bazelisk-linux-amd64
2025/03/03 05:23:55 Downloading https://releases.bazel.build/8.1.1/release/bazel-8.1.1-linux-x86_64...
Downloading: 58 MB out of 58 MB (100%)
WARNING: Invoking Bazel in batch mode since it is not invoked from within a workspace (below a directory having a MODULE.bazel file).
Extracting Bazel installation...
WARNING: Ignoring JAVA_HOME, because it must point to a JDK, not a JRE.
OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.ExceptionInInitializerError
at com.google.devtools.build.lib.skyframe.SkyframeExecutor.(SkyframeExecutor.java:449)
at com.google.devtools.build.lib.skyframe.BazelSkyframeExecutorConstants.newBazelSkyframeExecutorBuilder(BazelSkyframeExecutorConstants.java:45)
at com.google.devtools.build.lib.skyframe.SequencedSkyframeExecutorFactory.create(SequencedSkyframeExecutorFactory.java:44)
at com.google.devtools.build.lib.runtime.WorkspaceBuilder.build(WorkspaceBuilder.java:118)
at com.google.devtools.build.lib.runtime.BlazeRuntime.initWorkspace(BlazeRuntime.java:274)
at com.google.devtools.build.lib.runtime.BlazeRuntime.newRuntime(BlazeRuntime.java:1440)
at com.google.devtools.build.lib.runtime.BlazeRuntime.batchMain(BlazeRuntime.java:1031)
at com.google.devtools.build.lib.runtime.BlazeRuntime.main(BlazeRuntime.java:866)
at com.google.devtools.build.lib.bazel.Bazel.main(Bazel.java:104)
Caused by: java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Unknown Source)
at java.base/sun.nio.fs.UnixFileSystem.getPath(Unknown Source)
at java.base/java.nio.file.Path.of(Unknown Source)
at java.base/java.nio.file.Paths.get(Unknown Source)
at java.base/jdk.internal.platform.CgroupUtil.lambda$readStringValue$1(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/jdk.internal.platform.CgroupUtil.readStringValue(Unknown Source)
at java.base/jdk.internal.platform.CgroupSubsystemController.getStringValue(Unknown Source)
at java.base/jdk.internal.platform.CgroupSubsystemController.getLongValue(Unknown Source)
at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getLongValue(Unknown Source)
at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getHierarchical(Unknown Source)
at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.initSubSystem(Unknown Source)
at java.base/jdk.internal.platform.cgroupv1.CgroupV1Subsystem.getInstance(Unknown Source)
at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(Unknown Source)
at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(Unknown Source)
at java.base/jdk.internal.platform.CgroupMetrics.getInstance(Unknown Source)
at java.base/jdk.internal.platform.SystemMetrics.instance(Unknown Source)
at java.base/jdk.internal.platform.Metrics.systemMetrics(Unknown Source)
at java.base/jdk.internal.platform.Container.metrics(Unknown Source)
at jdk.management/com.sun.management.internal.OperatingSystemImpl.(Unknown Source)
at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean(Unknown Source)
at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap(Unknown Source)
at java.management/sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans(Unknown Source)
at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(Unknown Source)
at java.management/java.lang.management.ManagementFactory.getOperatingSystemMXBean(Unknown Source)
at com.google.devtools.build.lib.util.ResourceUsage.(ResourceUsage.java:46)
... 9 more
```
I can't figure out what the problem with the environment is. Any help would be appreciated.
EDIT: I used something like this to launch the slurm job.
```bash
srun \
--job-name=debug_ci_runner \
--time=1:00:00 \
--cpus-per-task=16 \
--gres=gpu:1 \
--container-image=/path/to/my/ci/images/gh_runner.sqsh \
--container-remap-root \
--no-container-mount-home \
--container-mounts=/usr/share/sunk:/usr/share/sunk \
--container-workdir=/actions-runner/ \
--pty /bin/bash
```
Contributor guide
Research direction
Start with the bazelisk-linux-amd64 reproducer and the shown srun container command, then trace the stack from Bazel's ResourceUsage initialization into Java's cgroup handling. Compare the behavior inside and outside the enroot/Slurm environment; done means identifying the environmental trigger and confirming a reliable fix or documented workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, go, java
- Domain
- build-system, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100