apache / apache/polaris

Management API Catalog Creation Fails with ContextNotActiveException (Quarkus CDI) in Polaris 1.1.0, 1.2.0 and 1.3.0 on K8

Open
#3,495 6 comments 0 reactions 0 assignees View on GitHub
bug stale
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

Hi Polaris team, can you help? Is there something basic I am missing or is this a known bug with a work-around recommended. Thanks in advance!

# Polaris Catalog Issue Report

## 1. What version of Apache Polaris are you using?
- **Version:** polaris-bin-1.1.0-incubating (image assembled from release archive) (but same issue occurred when using 1.2.0 and 1.3.0.
- **CLI binaries present:** `/opt/polaris/bin/admin` and `/opt/polaris/bin/server` are present; `/opt/polaris/bin/polaris` CLI binary is not present in the image.

## 2. What operating system and processor architecture are you using?
- **Environment:** Kubernetes (Rancher Desktop)
- **Namespace:** `flink-sandbox`
- **Operating System:** (Host OS not specified; running in containerized environment)
- **Processor Architecture:** (Not specified; typical x86_64 for containers)

## 3. What did you do?
- Integrated Polaris with Flink and MinIO on Kubernetes.
- Set up authentication using INTERNAL/default mode, token broker `rsa-key-pair`, and token refresh via CronJob.
- Attempted to create a catalog using the management API:
1. Obtained admin token via `/api/catalog/v1/oauth/tokens` (`scope=PRINCIPAL_ROLE:admin`, `grant_type=client_credentials`).
2. Called `POST /api/management/v1/catalogs` on port 8181 with valid token and (optionally) `X-Polaris-Realm: default-realm`.
3. Example request body:
```json
{
"catalog": {
"name": "warehouse",
"type": "INTERNAL",
"properties": {
"default-base-location": "s3a://warehouse/iceberg"
},
"storageConfigInfo": {
"storageType": "S3",
"allowedLocations": ["s3a://warehouse"]
}
}
}
```

## 4. What did you expect to see?
- Expected HTTP 201 response with catalog information.
- Expected successful end-to-end Flink ↔ Iceberg integration and passing integration tests.

## 5. What did you see instead?
- Received HTTP 500 error with `ContextNotActiveException` (Quarkus CDI) in server logs.
- Unable to create a catalog; integration tests blocked.
- Service exposes only port 8181; 8182 (health/management) is not exposed externally.
- CLI fallback unavailable (`/opt/polaris/bin/polaris` missing; `admin` tool lacks catalog commands).

---

## Additional Details

### Environment
- **Auth mode:** INTERNAL/default, token broker `rsa-key-pair`, max token TTL 1h.
- **Secrets:** RSA keys from `polaris-token-broker-keys`; admin credentials from `polaris-admin-credentials`.
- **Token refresh:** CronJob posts to `/api/catalog/v1/oauth/tokens` and writes access token to `polaris-token` secret (not mounted into Polaris container).
- **Storage/DB:** Postgres backend; MinIO as object store (s3a).

### Issue 1: Auth Behavior (resolved locally)
- Flink requests to Polaris REST catalog returned 401 even with `quarkus.oidc.enabled=false` and `POLARIS_AUTHENTICATION_ENABLED=false`.
- Workaround: Added env-based bearer token support in catalog manager ([pyflink/catalog/polaris_manager.py#L121-L137](pyflink/catalog/polaris_manager.py#L121-L137)). Token injected into Flink JM/TM as env `POLARIS_BEARER_TOKEN`.
- Result: Auth succeeded; subsequent failure was a 400 “Please specify a warehouse,” confirming token acceptance.

### Issue 2: Management API Catalog Create Fails (blocking)
- Symptom: `POST /api/management/v1/catalogs` returns HTTP 500 with `ContextNotActiveException` (Quarkus CDI).
- Impact: Cannot create a catalog; Flink ↔ Iceberg integration tests blocked.

### Hypothesis
- The error appears to be a server-side request context activation gap (e.g., missing `@ActivateRequestContext` or scope on a producer used by management/warehouse resources), based on the observed Quarkus CDI exception and referenced documentation.

### Requests for Community Guidance
1. Known issues in 1.1.0/1.3.0 where management APIs throw `ContextNotActiveException`? Any patches or backports?
2. Is there an official Polaris CLI binary for catalog operations that should ship in the binary distro?
3. Should the management API be called on 8182 instead of 8181? Expected port exposure and required headers?
4. Recommended workaround until a fix: is it safe to insert `@ActivateRequestContext` around management endpoints, or should the producer scope be adjusted elsewhere?

### Next-Step Options Considered
- Patch Polaris source (prefer 1.3.0) to wrap management/warehouse resources with request context and rebuild image (if source access is available).
- Expose 8182 and retest (to confirm if the error persists or to aid diagnosis).

### Artifacts and References
- Auth patch: `kubernetes/overlays/local/polaris-auth-patch.yaml`
- Token refresh CronJob: `kubernetes/overlays/local/polaris-token-refresh-cronjob.yaml`
- Deployment: `kubernetes/base/polaris/deployment.yaml`
- Service: `kubernetes/base/polaris/service.yaml`
- CLI binary absence confirmed via `kubectl exec deploy/polaris -- ls /opt/polaris/bin` (only `admin`, `server` present).

### Open Items
- Decode and share live token claims for validation (can provide if helpful).
- Confirm whether realm header is required beyond bearer token for management calls.

### To Reproduce

_No response_

### Actual Behavior

_No response_

### Expected Behavior

_No response_

### Additional context

_No response_

### System information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing POST /api/management/v1/catalogs with the documented token, realm header, and request body, then inspect the server logs for the ContextNotActiveException. Review kubernetes/base/polaris/deployment.yaml and service.yaml alongside the management API entry point; done means catalog creation returns HTTP 201 and the Flink–Iceberg integration is no longer blocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes, postgresql
Domain
api, backend, databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.