Error reading ORC files from Spark 2.4.* (ANTLR versions)
- Dominant language
- Jupyter Notebook
- Stars
- 2.9k
- Forks
- 383
- PR merge metrics
- No merged PRs in 30d
Description
# Description
It is not possible to read ORC files from Spark, at least using the docker image.
# Steps to reproduce the error:
1. Start docker instance:
```
docker run -p 8888:8888 \
--name jupyter \
beakerx/beakerx
```
2. Create a Scala notebook and try to create a orc file:
```
%classpath add mvn org.apache.spark spark-sql_2.11 2.4.5
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.appName("OSMDistribution")
.master("local[4]")
.config("spark.ui.enabled", "false")
.getOrCreate()
import spark.implicits._
spark.createDataset(1 to 1000).write.orc("/home/beakerx/data/output")
println(spark.read.orc("/home/beakerx/data/output").count)
spark.close()
```
3. Output
> ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.5ANTLR Runtime version 4.7 used for parser compilation does not match the current runtime version 4.5
```
java.lang.UnsupportedOperationException: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with UUID 59627784-3be5-417a-b9eb-8131a7286089 (expected aadb8d7e-aeef-4415-ad2b-8204d6cf042e or a legacy UUID).
at org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:154)
at org.apache.spark.sql.catalyst.parser.SqlBaseLexer.(SqlBaseLexer.java:1175)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:84)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:39)
at org.apache.spark.sql.execution.datasources.orc.OrcUtils$$anonfun$readSchema$1.applyOrElse(OrcUtils.scala:87)
at org.apache.spark.sql.execution.datasources.orc.OrcUtils$$anonfun$readSchema$1.applyOrElse(OrcUtils.scala:84)
at scala.collection.TraversableOnce$class.collectFirst(TraversableOnce.scala:145)
at scala.collection.AbstractIterator.collectFirst(Iterator.scala:1334)
at org.apache.spark.sql.execution.datasources.orc.OrcUtils$.readSchema(OrcUtils.scala:84)
at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.inferSchema(OrcFileFormat.scala:97)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
at scala.Option.orElse(Option.scala:289)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:179)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.orc(DataFrameReader.scala:666)
at org.apache.spark.sql.DataFrameReader.orc(DataFrameReader.scala:656)
... 46 elided
Caused by: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with UUID 59627784-3be5-417a-b9eb-8131a7286089 (expected aadb8d7e-aeef-4415-ad2b-8204d6cf042e or a legacy UUID).
... 65 more
```
# Notes
There are other issues related to mixing ANT versions in the classloader from different kernels. If this is the case, which docker image I must to use (the one using the same ANT version that Spark)?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.