googleapis / googleapis/java-pubsublite-spark

PubSubLite java.lang.NoSuchMethodError

Open
#637 3 comments 0 reactions 0 assignees View on GitHub
api: pubsublite
Dominant language
Java
Stars
18
Forks
14
PR merge metrics
No merged PRs in 30d

Description

I get NoSuchMethodError while trying to read message from pubsublite using spark structured streaming application.
```
2023-05-30 11:15:05,024 [INFO ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Reading table [com.google.cloud.pubsublite.spark.PslTable@79708f8] from DataSourceV2 named 'pubsublite' [com.google.cloud.pubsublite.spark.PslTableProvider@64ddf4ea]
2023-05-30 11:15:07,496 [INFO ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Starting new streaming query.
2023-05-30 11:15:07,498 [INFO ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Stream started from {}
2023-05-30 11:15:09,308 [INFO ] [84] org.apache.spark.sql.execution.streaming.CheckpointFileManager - Writing atomically to file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/0 using temp file file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/.0.40a4adab-d8ac-4e10-8af0-447fc3d8110c.tmp
2023-05-30 11:15:09,348 [INFO ] [84] org.apache.spark.sql.execution.streaming.CheckpointFileManager - Renamed temp file file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/.0.40a4adab-d8ac-4e10-8af0-447fc3d8110c.tmp to file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/0
2023-05-30 11:15:09,349 [INFO ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Committed offsets for batch 0. Metadata OffsetSeqMetadata(0,1685470509297,Map(spark.sql.streaming.stateStore.providerClass -> org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider, spark.sql.streaming.join.stateFormatVersion -> 2, spark.sql.streaming.stateStore.compression.codec -> lz4, spark.sql.streaming.stateStore.rocksdb.formatVersion -> 5, spark.sql.streaming.statefulOperator.useStrictDistribution -> true, spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion -> 2, spark.sql.streaming.multipleWatermarkPolicy -> min, spark.sql.streaming.aggregation.stateFormatVersion -> 2, spark.sql.shuffle.partitions -> 200))

Exception in thread "pubsub-lite-futures-1" java.lang.NoSuchMethodError: 'com.google.api.gax.rpc.ApiCallContext com.google.cloud.pubsublite.internal.wire.ServiceClients.getCallContext(com.google.cloud.pubsublite.internal.wire.PubsubContext, com.google.cloud.pubsublite.internal.wire.RoutingMetadata)'
at com.google.cloud.pubsublite.spark.PslReadDataSourceOptions.lambda$null$1(PslReadDataSourceOptions.java:139)
at com.google.cloud.pubsublite.internal.wire.SingleConnection.(SingleConnection.java:71)
at com.google.cloud.pubsublite.internal.wire.SingleConnection.(SingleConnection.java:77)
at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl.(ConnectedSubscriberImpl.java:54)
at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl.(ConnectedSubscriberImpl.java:33)
at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl$Factory.New(ConnectedSubscriberImpl.java:46)
at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl$Factory.New(ConnectedSubscriberImpl.java:39)
at com.google.cloud.pubsublite.internal.wire.RetryingConnectionImpl.reinitialize(RetryingConnectionImpl.java:106)
at com.google.cloud.pubsublite.internal.wire.RetryingConnectionImpl.lambda$doStart$0(RetryingConnectionImpl.java:94)
```

i do see the getCallContext() method with required parameter and classes are present at package ` com.google.cloud.pubsublite.internal.wire` which is available in google-cloud-pubsublite-1.12.5 jar

I also noticed another thing Upon looking at the `PartitionSubscriberFactory getSubscriberFactory()` method in `com.google.cloud.pubsublite.spark.PslReadDataSourceOptions` it indeed using different consumer
`com.google.cloud.pubsublite.SequencedMessage` vs setMessageConsumer expecting ` com.google.cloud.pubsublite.proto.SequencedMessage`

#### Environment details

1. Specify the API at the beginning of the title. For example, "BigQuery: ...") : **PubSub Lite**
General, Core, and Other are also allowed as types
2. OS type and version: Mac OS M1(ventura)
3. Java version: **11**
4. version(s):

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.