apache / apache/polaris

Polaris crash-loops on local Kubernetes: the OTel GCP resource detector blocks startup for ~135s

Open
#5,401 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

#### Summary
Where `metadata.google.internal` resolves to an address that silently drops packets, the `main` thread blocks ~135s in a socket `connect()` to the GCP metadata server, from an OpenTelemetry resource detector run during Quarkus startup. Nothing binds 8181/8182 until it returns.

This happens **even with Polaris's shipped `quarkus.otel.sdk.disabled=true` default**: OTel autoconfigure runs resource providers before honoring that flag.

A bare pod recovers after the stall. Under the Helm chart it does not — the default liveness probe kills the container ~60s in, so the pod crash-loops indefinitely. There is no error and no stack trace, just an empty pod log.

### To Reproduce

#### Reproduced on
* `main` 4b4218638 (Quarkus 3.38.3, JDK 21) and published apache/polaris:1.7.0
* kind, k3d/k3s, minikube, OrbStack Kubernetes, and plain `docker run`
* Not reproducible on Amazon EKS — see below

#### Steps to reproduce
```shell
docker run --rm apache/polaris:1.7.0
```

### Actual Behavior

No `started in` line for ~135s.

#### Measured
Same image (`apache/polaris:1.7.0`); "startup" is Quarkus's own `started in` figure, excluding image pull.

| Environment | `metadata.google.internal` | `169.254.169.254` | Startup |
| --- | --- | --- | --- |
| Amazon EKS v1.34.9 | **NXDOMAIN** | live IMDS (`401`/`200`, `connect=0.0002s`) | **2.772s** |
| kind v0.32.0 | 169.254.169.254 | no response | **134.326s** |
| k3d v5.9.0 / k3s v1.35.5 | 169.254.169.254 | no response | **135.607s** |
| minikube v1.38.1 (docker) | 169.254.169.254 | no response | **137.166s** |
| OrbStack Kubernetes v1.35.6 | 169.254.169.254 | no response | **136.217s** |
| OrbStack Docker | 169.254.169.254 | no response | **137.740s** |

The affected environments agree at ~134-138s: a TCP connect timeout, not an unbounded hang.

EKS is protected twice over. `getent hosts metadata.google.internal` returns NXDOMAIN (`.internal` is reserved and never publicly delegated), and even if it resolved, `169.254.169.254` there is the live IMDS endpoint answering in 0.2ms. The stall needs a *silent drop*, not merely an absent metadata service. Running the same image with `--dns 1.1.1.1` locally confirms the NXDOMAIN path: 1.589s.

At risk, then: local/VM Kubernetes and container runtimes that map the metadata name to an unreachable link-local address, plus split-horizon or wildcard DNS that returns a sinkhole A record instead of NXDOMAIN.

#### Impact under the helm chart
The chart's `livenessProbe` (`/q/health/live`, `initialDelaySeconds: 5`, `periodSeconds: 10`, `failureThreshold: 3`) starts probing while nothing is listening. Deployed with those settings, the pod restarted 5 times in 5 minutes, never healthy — `failed liveness probe, will be restarted`, `connection refused` on 8182, `exitCode=137`. It gets ~60s and needs ~135s, so it can never win that race.

### Expected Behavior

`started in` should show up within ~1.5s.

### Additional context

#### Root cause
SIGQUIT thread dump of the stalled `main` thread:
```
sun.nio.ch.Net.connect0(Native Method)
...
com.google.cloud.opentelemetry.detection.GCPMetadataConfig.fetchAttribute(GCPMetadataConfig.java:155)
com.google.cloud.opentelemetry.detection.GCPMetadataConfig.getProjectId(GCPMetadataConfig.java:52)
com.google.cloud.opentelemetry.detection.GCPPlatformDetector.isRunningOnGcp(GCPPlatformDetector.java:65)
io.opentelemetry.contrib.gcp.resource.GCPResourceProvider.getAttributes(GCPResourceProvider.java:88)
io.opentelemetry.sdk.autoconfigure.ResourceConfiguration.configureResource(ResourceConfiguration.java:81)
io.quarkus.opentelemetry.runtime.OpenTelemetryRecorder$3.apply(OpenTelemetryRecorder.java:116)
```

`GCPMetadataConfig` hardcodes `http://metadata.google.internal/computeMetadata/v1/` with no connect timeout, so two conditions must hold together:

1. `metadata.google.internal` resolves, **and**
2. that address silently drops packets, so `connect()` blocks for the full TCP
SYN-retry budget instead of failing.

That is why it reproduces in some environments and not others.

#### Why it reaches the classpath
Not a direct dependency — it arrives transitively with GCS support:
```
:polaris-runtime-service -> :polaris-core
-> com.google.cloud:google-cloud-storage:2.70.0
-> io.opentelemetry.contrib:opentelemetry-gcp-resources:1.37.0-alpha
-> com.google.cloud.opentelemetry:detector-resources-support:0.33.0
```
`opentelemetry-gcp-resources` registers `GCPResourceProvider` in `META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.ResourceProvider`, so autoconfigure discovers and runs it. Nothing in Polaris opts in. No AWS resource detectors are on the classpath, so only the GCP one is implicated.

#### Workaround
```
OTEL_JAVA_DISABLED_RESOURCE_PROVIDERS=io.opentelemetry.contrib.gcp.resource.GCPResourceProvider
```
Same build: `started in 1.809s`.

#### Suggested fix

Production cloud deployments (EKS/EC2, and GCP itself) are unaffected, so this likely does not warrant changing behaviour for everyone.

1. **Document it.** The main ask. The symptom offers nothing to search for. A short note in the Helm / getting-started docs — symptom, the `GCPMetadataConfig.fetchAttribute` thread-dump signature, and the env var above — would make it self-service.

2. **A chart toggle.** `tracing.enabled: false` (the default) is already the branch setting `quarkus.otel.sdk.disabled=true` in `helm/polaris/templates/configmap.yaml:211`. Either have that branch also disable the cloud resource providers (an operator who turned tracing off never wants a metadata probe blocking startup), or expose `tracing.disabledResourceProviders` for those who enable tracing but run in an affected environment. No strong preference.

**Not suggested:** disabling `GCPResourceProvider` globally in `runtime/defaults` — that costs resource attributes for legitimate GCP users to fix a sinkhole-DNS problem.

Happy to open a PR if maintainers agree on the approach.

### System information

OS: macOS 26.2 (arm64) — host for every local environment below
Polaris Catalog Version: apache/polaris:1.7.0 (published image), and main 4b4218638 built locally (polaris-server 1.7.0-SNAPSHOT)
Object storage & setup: none — default in-memory persistence, no catalog or object storage configured. The bug occurs during Quarkus startup, before any storage is touched.

Runtime detail, since the trigger is environmental rather than configuration:

JVM: OpenJDK 21.0.11 (in image) / 21.0.12 (local build)
Quarkus: 3.37.4 (image) / 3.38.3 (main @ 4b4218638)
Container runtime: OrbStack 29.4.0 (Docker-compatible)
Affected: kind v0.32.0, k3d v5.9.0 / k3s v1.35.5,
minikube v1.38.1 (docker driver),
OrbStack Kubernetes v1.35.6, plain `docker run`
Not affected: Amazon EKS v1.34.9

The decisive variable is DNS, not the Polaris configuration: whether `metadata.google.internal` resolves, and whether that address answers.

Contributor guide

Open the contributing guide

Research direction

Locate the relevant Helm or getting-started documentation and inspect helm/polaris/templates/configmap.yaml around line 211 to understand the existing tracing setting. Document the startup stall, the GCPMetadataConfig.fetchAttribute thread-dump signature, and the OTEL_JAVA_DISABLED_RESOURCE_PROVIDERS workaround; done means a local Kubernetes user can identify and avoid the crash loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, helm, java, kubernetes
Domain
devops, documentation, infrastructure
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.