SparkStructureStreamingRunner issue due to spark3.*
Open
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
We are getting below error on using spark 3 and above version:
java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/DataSourceV2
This error is due to SparkStructuredStreamingRunner internally using DataSourceV2 api from spark 3.* this class is not found, as it is tightly coupled with DataSourceV2 class we are facing this issue:
Please see below code snippet for the same.
public class DatasetSourceBatch implements DataSourceV2, ReadSupport {
@Override
public DataSourceReader createReader(DataSourceOptions options)
{ return new DatasetReader<>(options); }
Contributor guide
Assessment
This issue has not been assessed yet.