Bug: `java.lang.NoSuchMethodError: 'void org.rocksdb.WriteBatchWithIndex.remove(byte[])'`

Hi there,

I have begun getting errors that cause my EMR Serverless Spark Structured Streaming job to fail after I __cancelled__ the job and __restarted__ it.

```
24/09/03 15:01:46 ERROR MicroBatchExecution: Query [id = c14e284a-cef1-469c-9df0-906e51a7a88c, runId = 27373269-f960-41f3-a54b-bfe228f869fb] terminated with error
java.lang.NoSuchMethodError: 'void org.rocksdb.WriteBatchWithIndex.remove(byte[])'
```
and
```
Exception in thread "stream execution thread for [id = c14e284a-cef1-469c-9df0-906e51a7a88c, runId = f928d571-456a-4b30-92a6-7691b9e6f298]" java.lang.NoSuchMethodError: 'void org.rocksdb.WriteBatchWithIndex.remove(byte[])'
```

It seems that when new data arrives in the bucket the job will fail. The job restarts on a retry and then does process data in the S3 bucket and clear the queue.

My Spark properties are configured as follows:
```
--conf spark.jars=https://awslabs-code-us-east-1.s3.amazonaws.com/spark-streaming-sql-s3-connector/spark-streaming-sql-s3-connector-0.0.2.jar
--conf spark.jars.packages=org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2,org.apache.iceberg:iceberg-aws-bundle:1.5.2,software.amazon.awssdk:bundle:2.25.56
--conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.glue_catalog.warehouse=s3://
--conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
--conf spark.sql.streaming.stateStore.providerClass=org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider
--conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
--conf spark.driver.cores=1
--conf spark.driver.memory=2g
--conf spark.dynamicAllocation.enabled=true
--conf spark.dynamicAllocation.executorIdleTimeout=20s
--conf spark.executor.cores=1
--conf spark.executor.memory=2g
```

Clearing temp files resolves the issue but that shouldn't be required.

Start with the EMR Serverless Spark Structured Streaming configuration shown in the issue, especially the connector JAR, package versions, and RocksDB state-store provider. Reproduce cancellation, restart, and new S3 data arrival; done means the job processes the data without the NoSuchMethodError or requiring temporary-file cleanup.

Written by the indexing model from the issue text.