Polaris crash-loops on local Kubernetes: the OTel GCP resource detector blocks startup for ~135s
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Describe the bug
#### Summary
Where `metadata.google.internal` resolves to an address that silently drops packets, the `main` thread blocks ~135s in a socket `connect()` to the GCP metadata server, from an OpenTelemetry resource detector run during Quarkus startup. Nothing binds 8181/8182 until it returns.
This happens **even with Polaris's shipped `quarkus.otel.sdk.disabled=true` default**: OTel autoconfigure runs resource providers before honoring that flag.
A bare pod recovers after the stall. Under the Helm chart it does not — the default liveness probe kills the container ~60s in, so the pod crash-loops indefinitely. There is no error and no stack trace, just an empty pod log.
### To Reproduce
#### Reproduced on
* `main` 4b4218638 (Quarkus 3.38.3, JDK 21) and published apache/polaris:1.7.0
* kind, k3d/k3s, minikube, OrbStack Kubernetes, and plain `docker run`
* Not reproducible on Amazon EKS — see below
#### Steps to reproduce
```shell
docker run --rm apache/polaris:1.7.0
```
### Actual Behavior
No `started in` line for ~135s.
#### Measured
Same image (`apache/polaris:1.7.0`); "startup" is Quarkus's own `started in` figure, excluding image pull.
| Environment | `metadata.google.internal` | `169.254.169.254` | Startup |
| --- | --- | --- | --- |
| Amazon EKS v1.34.9 | **NXDOMAIN** | live IMDS (`401`/`200`, `connect=0.0002s`) | **2.772s** |
| kind v0.32.0 | 169.254.169.254 | no response | **134.326s** |
| k3d v5.9.0 / k3s v1.35.5 | 169.254.169.254 | no response | **135.607s** |
| minikube v1.38.1 (docker) | 169.254.169.254 | no response | **137.166s** |
| OrbStack Kubernetes v1.35.6 | 169.254.169.254 | no response | **136.217s** |
| OrbStack Docker | 169.254.169.254 | no response | **137.740s** |
The affected environments agree at ~134-138s: a TCP connect timeout, not an unbounded hang.
EKS is protected twice over. `getent hosts metadata.google.internal` returns NXDOMAIN (`.internal` is reserved and never publicly delegated), and even if it resolved, `169.254.169.254` there is the live IMDS endpoint answering in 0.2ms. The stall needs a *silent drop*, not merely an absent metadata service. Running the same image with `--dns 1.1.1.1` locally confirms the NXDOMAIN path: 1.589s.
At risk, then: local/VM Kubernetes and container runtimes that map the metadata name to an unreachable link-local address, plus split-horizon or wildcard DNS that returns a sinkhole A record instead of NXDOMAIN.
#### Impact under the helm chart
The chart's `livenessProbe` (`/q/health/live`, `initialDelaySeconds: 5`, `periodSeconds: 10`, `failureThreshold: 3`) starts probing while nothing is listening. Deployed with those settings, the pod restarted 5 times in 5 minutes, never healthy — `failed liveness probe, will be restarted`, `connection refused` on 8182, `exitCode=137`. It gets ~60s and needs ~135s, so it can never win that race.
### Expected Behavior
`started in` should show up within ~1.5s.
### Additional context
#### Root cause
SIGQUIT thread dump of the stalled `main` thread:
```
sun.nio.ch.Net.connect0(Native Method)
...
com.google.cloud.opentelemetry.detection.GCPMetadataConfig.fetchAttribute(GCPMetadataConfig.java:155)
com.google.cloud.opentelemetry.detection.GCPMetadataConfig.getProjectId(GCPMetadataConfig.java:52)
com.google.cloud.opentelemetry.detection.GCPPlatformDetector.isRunningOnGcp(GCPPlatformDetector.java:65)
io.opentelemetry.contrib.gcp.resource.GCPResourceProvider.getAttributes(GCPResourceProvider.java:88)
io.opentelemetry.sdk.autoconfigure.ResourceConfiguration.configureResource(ResourceConfiguration.java:81)
io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$3.apply(OpenTelemetryRecorder.java:116)
```
`GCPMetadataConfig` hardcodes `http://metadata.google.internal/computeMetadata/v1/` with no connect timeout, so two conditions must hold together:
1. `metadata.google.internal` resolves, **and**
2. that address silently drops packets, so `connect()` blocks for the full TCP
SYN-retry budget instead of failing.
That is why it reproduces in some environments and not others.
#### Why it reaches the classpath
Not a direct dependency — it arrives transitively with GCS support:
```
:polaris-runtime-service -> :polaris-core
-> com.google.cloud:google-cloud-storage:2.70.0
-> io.opentelemetry.contrib:opentelemetry-gcp-resources:1.37.0-alpha
-> com.google.cloud.opentelemetry:detector-resources-support:0.33.0
```
`opentelemetry-gcp-resources` registers `GCPResourceProvider` in `META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.ResourceProvider`, so autoconfigure discovers and runs it. Nothing in Polaris opts in. No AWS resource detectors are on the classpath, so only the GCP one is implicated.
#### Workaround
```
OTEL_JAVA_DISABLED_RESOURCE_PROVIDERS=io.opentelemetry.contrib.gcp.resource.GCPResourceProvider
```
Same build: `started in 1.809s`.
#### Suggested fix
Production cloud deployments (EKS/EC2, and GCP itself) are unaffected, so this likely does not warrant changing behaviour for everyone.
1. **Document it.** The main ask. The symptom offers nothing to search for. A short note in the Helm / getting-started docs — symptom, the `GCPMetadataConfig.fetchAttribute` thread-dump signature, and the env var above — would make it self-service.
2. **A chart toggle.** `tracing.enabled: false` (the default) is already the branch setting `quarkus.otel.sdk.disabled=true` in `helm/polaris/templates/configmap.yaml:211`. Either have that branch also disable the cloud resource providers (an operator who turned tracing off never wants a metadata probe blocking startup), or expose `tracing.disabledResourceProviders` for those who enable tracing but run in an affected environment. No strong preference.
**Not suggested:** disabling `GCPResourceProvider` globally in `runtime/defaults` — that costs resource attributes for legitimate GCP users to fix a sinkhole-DNS problem.
Happy to open a PR if maintainers agree on the approach.
### System information
OS: macOS 26.2 (arm64) — host for every local environment below
Polaris Catalog Version: apache/polaris:1.7.0 (published image), and main 4b4218638 built locally (polaris-server 1.7.0-SNAPSHOT)
Object storage & setup: none — default in-memory persistence, no catalog or object storage configured. The bug occurs during Quarkus startup, before any storage is touched.
Runtime detail, since the trigger is environmental rather than configuration:
JVM: OpenJDK 21.0.11 (in image) / 21.0.12 (local build)
Quarkus: 3.37.4 (image) / 3.38.3 (main @ 4b4218638)
Container runtime: OrbStack 29.4.0 (Docker-compatible)
Affected: kind v0.32.0, k3d v5.9.0 / k3s v1.35.5,
minikube v1.38.1 (docker driver),
OrbStack Kubernetes v1.35.6, plain `docker run`
Not affected: Amazon EKS v1.34.9
The decisive variable is DNS, not the Polaris configuration: whether `metadata.google.internal` resolves, and whether that address answers.
Contributor guide
Research direction
Locate the relevant Helm or getting-started documentation and inspect helm/polaris/templates/configmap.yaml around line 211 to understand the existing tracing setting. Document the startup stall, the GCPMetadataConfig.fetchAttribute thread-dump signature, and the OTEL_JAVA_DISABLED_RESOURCE_PROVIDERS workaround; done means a local Kubernetes user can identify and avoid the crash loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, helm, java, kubernetes
- Domain
- devops, documentation, infrastructure
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100