googleapis / googleapis/java-pubsublite-spark

Fails pulling topic in dataproc 2.2 with PERMISSION_DENIED error

Open
#659 0 comments 0 reactions 0 assignees View on GitHub
api: pubsublite
Dominant language
Java
Stars
18
Forks
14
PR merge metrics
No merged PRs in 30d

Description

Unable to read for pubsublite topic in dataproc 2.2

#### Environment details

1. Specify the API at the beginning of the title. For example, "BigQuery: ...").
General, Core, and Other are also allowed as types
2. OS type and version:
3. Dataproc version: 2.2.0-RC2-debian11
4. version(s):
Spark: 3.4
Scala: 2.12.18
Iceberg: 1.4
pubsublite-spark-sql-streaming: 1.0.0
biglake: 0.1.0

pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar
iceberg-spark-runtime-3.4_2.12-1.4.0.jar
biglake-catalog-iceberg1.2.0-0.1.0-with-dependencies.jar

#### Code example

```scala
implicit val pubSubSource: StreamingSource[PubSubSource] = new StreamingSource[PubSubSource] {
override def streamingSource(conf: PubSubSource)(implicit spark: SparkSession): DataFrame =
spark
.readStream
.format("pubsublite")
.option(
"pubsublite.subscription",
conf.subscriptionName
)
.load()
```

#### Stack trace
```
Caused by: java.util.concurrent.ExecutionException: com.google.api.gax.rpc.PermissionDeniedException: com.google.cloud.pubsublite.repackaged.io.grpc.StatusRuntimeException: PERMISSION_DENIED: Request had insufficient authentication scopes.
at repackaged.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
at repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:567) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
at com.google.cloud.pubsublite.spark.PslReadDataSourceOptions.getTopicPath(PslReadDataSourceOptions.java:207) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
... 54 more
Caused by: com.google.api.gax.rpc.PermissionDeniedException: com.google.cloud.pubsublite.repackaged.io.grpc.StatusRuntimeException: PERMISSION_DENIED: Request had insufficient authentication scopes.
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:98) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66) ~[pubsublite-spark-sql-streaming-1.0.0-with-dependencies.jar:?]
```

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.