apache / apache/iceberg

GoogleAuthManager - Invalid credentials endpoint: null, for REST credentials vending mode

Open
#14,982 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.6.1

### Query engine

Spark

### Please describe the bug 🐞

Hey,

We're experimenting setting up an Iceberg-based data lakehouse on BigLake Metastore. We tried to connect Dataproc Spark to BigLake via REST catalog endpoint, but unfortunately it only partially works... We use vended credentials mode.

Spark: 3.5.3, Dataproc: 2.2.38, Iceberg: 1.6.1, Iceberg GCP bundle: 1.6.1.
"org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,org.apache.iceberg:iceberg-gcp-bundle:1.6.1",

```
spark = (
SparkSession.builder.appName("biglake")
.config(
"spark.jars.packages",
"org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,org.apache.iceberg:iceberg-gcp-bundle:1.6.1",
)
.config(
f"spark.sql.catalog.{catalog_name}", "org.apache.iceberg.spark.SparkCatalog"
)
.config(f"spark.sql.catalog.{catalog_name}.type", "rest")
.config(
f"spark.sql.catalog.{catalog_name}.uri",
"https://biglake.googleapis.com/iceberg/v1/restcatalog",
)
.config(f"spark.sql.catalog.{catalog_name}.warehouse", "gs://{project}")
.config(
f"spark.sql.catalog.{catalog_name}.header.x-goog-user-project", project
)
.config(
f"spark.sql.catalog.{catalog_name}.oauth2-server-uri",
"https://biglake.googleapis.com/iceberg/v1/restcatalog/v1/oauth/tokens",
)
.config(
f"spark.sql.catalog.{catalog_name}.rest.auth.type",
"org.apache.iceberg.gcp.auth.GoogleAuthManager",
)
.config(
f"spark.sql.catalog.{catalog_name}.io-impl",
"org.apache.iceberg.gcp.gcs.GCSFileIO",
)
.config(
f"spark.sql.catalog.{catalog_name}.header.X-Iceberg-Access-Delegation",
"vended-credentials",
)
.config(f"spark.sql.catalog.{catalog_name}.rest-metrics-reporting-enabled", "false")
.config(
"spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
)
.config("spark.sql.defaultCatalog", "biglake")
.getOrCreate()
)
```

Metadata (catalog) exploration works (makes us think auth eventually succeeds), but immediately fails when querying actual data e.g. `df = spark.sql("SELECT count(*) FROM feature_data.nodes")` with `Invalid credentials endpoint: null`.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.