Azure / Azure/azure-sdk-for-java

[BUG][Spark Connector]When using FullFidelity change feed mode, the spark job failed with NullPointerException

Open
#43,126 2 comments 1 reaction 0 assignees View on GitHub
Client Cosmos cosmos-java-ecosystem-se-planning cosmos:spark3 ffcf-ga
Dominant language
Java
Stars
2.6k
Forks
2.2k
Avg merge
2d 8h
Merged PRs (30d)
178

Description

## Issue:
```
Job aborted due to stage failure: Task 3 in stage 22.0 failed 1 times, most recent failure: Lost task 3.0 in stage 22.0 (TID 79) ```
(d154638bc000009.abqsg4x0r5nu5avvstxro4uyoe.dx.internal.cloudapp.net executor driver): java.lang.NullPointerException
at com.azure.cosmos.spark.CosmosRowConverterBase.parseId(CosmosRowConverterBase.scala:708)
at com.azure.cosmos.spark.CosmosRowConverterBase.$anonfun$convertStructToChangeFeedSparkDataTypeV1$1(CosmosRowConverterBase.scala:835)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at com.azure.cosmos.spark.CosmosRowConverterBase.convertStructToChangeFeedSparkDataTypeV1(CosmosRowConverterBase.scala:831)
at com.azure.cosmos.spark.CosmosRowConverterBase.fromObjectNodeToChangeFeedRowV1(CosmosRowConverterBase.scala:93)
at com.azure.cosmos.spark.ChangeFeedPartitionReader.com$azure$cosmos$spark$ChangeFeedPartitionReader$$changeFeedItemFactoryMethodV1(ChangeFeedPartitionReader.scala:110)
at com.azure.cosmos.spark.ChangeFeedPartitionReader$$anon$2.deserialize(ChangeFeedPartitionReader.scala:156)
at com.azure.cosmos.CosmosItemSerializer.deserializeSafe(CosmosItemSerializer.java:95)
at com.azure.cosmos.CosmosItemSerializer$1.deserializeSafe(CosmosItemSerializer.java:212)
at com.azure.cosmos.implementation.Utils.parse(Utils.java:645)
at com.azure.cosmos.implementation.RxDocumentServiceResponse.getQueryResponse(RxDocumentServiceResponse.java:188)
```

## Possible root cause:
The issue happens for delete operation, when current and previous node are both being null (whether the previous image will be returned could be controlled by a backend flag config)

## Fix
* Follow up when the previous image will be returned
* Fix in spark connector code to check whether previous image exists, and handle null cases correctly (for example return null for the properties need to be parsed)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.