Azure / Azure/azure-sdk-for-java
[BUG][Spark Connector]When using FullFidelity change feed mode, the spark job failed with NullPointerException
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 2.2k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 178
Description
## Issue:
```
Job aborted due to stage failure: Task 3 in stage 22.0 failed 1 times, most recent failure: Lost task 3.0 in stage 22.0 (TID 79) ```
(d154638bc000009.abqsg4x0r5nu5avvstxro4uyoe.dx.internal.cloudapp.net executor driver): java.lang.NullPointerException
at com.azure.cosmos.spark.CosmosRowConverterBase.parseId(CosmosRowConverterBase.scala:708)
at com.azure.cosmos.spark.CosmosRowConverterBase.$anonfun$convertStructToChangeFeedSparkDataTypeV1$1(CosmosRowConverterBase.scala:835)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at com.azure.cosmos.spark.CosmosRowConverterBase.convertStructToChangeFeedSparkDataTypeV1(CosmosRowConverterBase.scala:831)
at com.azure.cosmos.spark.CosmosRowConverterBase.fromObjectNodeToChangeFeedRowV1(CosmosRowConverterBase.scala:93)
at com.azure.cosmos.spark.ChangeFeedPartitionReader.com$azure$cosmos$spark$ChangeFeedPartitionReader$$changeFeedItemFactoryMethodV1(ChangeFeedPartitionReader.scala:110)
at com.azure.cosmos.spark.ChangeFeedPartitionReader$$anon$2.deserialize(ChangeFeedPartitionReader.scala:156)
at com.azure.cosmos.CosmosItemSerializer.deserializeSafe(CosmosItemSerializer.java:95)
at com.azure.cosmos.CosmosItemSerializer$1.deserializeSafe(CosmosItemSerializer.java:212)
at com.azure.cosmos.implementation.Utils.parse(Utils.java:645)
at com.azure.cosmos.implementation.RxDocumentServiceResponse.getQueryResponse(RxDocumentServiceResponse.java:188)
```
## Possible root cause:
The issue happens for delete operation, when current and previous node are both being null (whether the previous image will be returned could be controlled by a backend flag config)
## Fix
* Follow up when the previous image will be returned
* Fix in spark connector code to check whether previous image exists, and handle null cases correctly (for example return null for the properties need to be parsed)
Contributor guide
Assessment
This issue has not been assessed yet.