googleapis / googleapis/java-pubsublite

[pubsublite] incompatibility between google-cloud-pubsublite and pubsublite-spark-sql-streaming

Aperta
#1,453 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
api: pubsublite
Lingua principale
Java
Stelle
42
Fork
27
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

1. Specify the API at the beginning of the title. For example, "BigQuery: ...").
General, Core, and Other are also allowed as types
2. OS type and version: Mac M1
3. Java version: 11
4. version(s):

#### Steps to reproduce

1. try to create pubsub reader (same as examples from repo) using google-cloud-pubsublite:1.12.10 ( any version after 1.10) and pubsublite-spark-sql-streaming: 1.0.0

#### Code example

```java
// example from https://github.com/googleapis/java-pubsublite-spark/tree/main/samples
```

#### Stack trace
```
2023-05-30 11:15:05,024 [INFO ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Reading table [com.google.cloud.pubsublite.spark.PslTable@79708f8] from DataSourceV2 named 'pubsublite' [com.google.cloud.pubsublite.spark.PslTableProvider@64ddf4ea]
2023-05-30 11:15:07,496 [INFO ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Starting new streaming query.
2023-05-30 11:15:07,498 [INFO ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Stream started from {}
2023-05-30 11:15:09,308 [INFO ] [84] org.apache.spark.sql.execution.streaming.CheckpointFileManager - Writing atomically to file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/0 using temp file file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/.0.40a4adab-d8ac-4e10-8af0-447fc3d8110c.tmp
2023-05-30 11:15:09,348 [INFO ] [84] org.apache.spark.sql.execution.streaming.CheckpointFileManager - Renamed temp file file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/.0.40a4adab-d8ac-4e10-8af0-447fc3d8110c.tmp to file:/private/var/folders/m9/fgqzm44d0111yh6kpgv51wpw0000gn/T/temporary-a794d805-5113-4f16-a758-36d8ce45a5e8/offsets/0
2023-05-30 11:15:09,349 [INFO ] [84] org.apache.spark.sql.execution.streaming.MicroBatchExecution - Committed offsets for batch 0. Metadata OffsetSeqMetadata(0,1685470509297,Map(spark.sql.streaming.stateStore.providerClass -> org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider, spark.sql.streaming.join.stateFormatVersion -> 2, spark.sql.streaming.stateStore.compression.codec -> lz4, spark.sql.streaming.stateStore.rocksdb.formatVersion -> 5, spark.sql.streaming.statefulOperator.useStrictDistribution -> true, spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion -> 2, spark.sql.streaming.multipleWatermarkPolicy -> min, spark.sql.streaming.aggregation.stateFormatVersion -> 2, spark.sql.shuffle.partitions -> 200))

Exception in thread "pubsub-lite-futures-1" java.lang.NoSuchMethodError: 'com.google.api.gax.rpc.ApiCallContext com.google.cloud.pubsublite.internal.wire.ServiceClients.getCallContext(com.google.cloud.pubsublite.internal.wire.PubsubContext, com.google.cloud.pubsublite.internal.wire.RoutingMetadata)'
at com.google.cloud.pubsublite.spark.PslReadDataSourceOptions.lambda$null$1(PslReadDataSourceOptions.java:139)
at com.google.cloud.pubsublite.internal.wire.SingleConnection.(SingleConnection.java:71)
at com.google.cloud.pubsublite.internal.wire.SingleConnection.(SingleConnection.java:77)
at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl.(ConnectedSubscriberImpl.java:54)
at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl.(ConnectedSubscriberImpl.java:33)
at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl$Factory.New(ConnectedSubscriberImpl.java:46)
at com.google.cloud.pubsublite.internal.wire.ConnectedSubscriberImpl$Factory.New(ConnectedSubscriberImpl.java:39)
at com.google.cloud.pubsublite.internal.wire.RetryingConnectionImpl.reinitialize(RetryingConnectionImpl.java:106)
at com.google.cloud.pubsublite.internal.wire.RetryingConnectionImpl.lambda$doStart$0(RetryingConnectionImpl.java:94)
```

#### External references such as API reference guides

- ?

#### Any additional information below
ref:
https://github.com/googleapis/java-pubsublite-spark/issues/637
https://github.com/googleapis/java-pubsublite/pull/1335

Thanks!

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia con l’esempio referenziato in java-pubsublite-spark e l’errore in PslReadDataSourceOptions.java:139. Confronta google-cloud-pubsublite 1.12.10 con pubsublite-spark-sql-streaming 1.0.0, quindi leggi l’issue 637 e la pull request 1335 collegate per il contesto di compatibilità. Il lavoro è completato quando il reader si avvia senza il NoSuchMethodError segnalato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
java, spark
Ambito
data-engineering, stream-processing
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.