apache / apache/beam

SparkStructureStreamingRunner issue due to spark3.*

Open
#22,492 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

We are getting below error on using spark 3 and above version:

java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/DataSourceV2

This error is due to SparkStructuredStreamingRunner internally using DataSourceV2 api from spark 3.* this class is not found, as it is tightly coupled with DataSourceV2 class we are facing this issue:

Please see below code snippet for the same.

public class DatasetSourceBatch implements DataSourceV2, ReadSupport {

@Override
public DataSourceReader createReader(DataSourceOptions options)

{ return new DatasetReader<>(options); }

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.