twosigma / twosigma/beakerx

Error reading ORC files from Spark 2.4.* (ANTLR versions)

Open
#8,281 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.9k
Forks
383
PR merge metrics
No merged PRs in 30d

Description

# Description
It is not possible to read ORC files from Spark, at least using the docker image.

# Steps to reproduce the error:

1. Start docker instance:
```
docker run -p 8888:8888 \
--name jupyter \
beakerx/beakerx
```

2. Create a Scala notebook and try to create a orc file:
```
%classpath add mvn org.apache.spark spark-sql_2.11 2.4.5

import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.appName("OSMDistribution")
.master("local[4]")
.config("spark.ui.enabled", "false")
.getOrCreate()

import spark.implicits._
spark.createDataset(1 to 1000).write.orc("/home/beakerx/data/output")
println(spark.read.orc("/home/beakerx/data/output").count)
spark.close()
```

3. Output
> ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.5ANTLR Runtime version 4.7 used for parser compilation does not match the current runtime version 4.5

```
java.lang.UnsupportedOperationException: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with UUID 59627784-3be5-417a-b9eb-8131a7286089 (expected aadb8d7e-aeef-4415-ad2b-8204d6cf042e or a legacy UUID).
at org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:154)
at org.apache.spark.sql.catalyst.parser.SqlBaseLexer.(SqlBaseLexer.java:1175)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:84)
at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:39)
at org.apache.spark.sql.execution.datasources.orc.OrcUtils$$anonfun$readSchema$1.applyOrElse(OrcUtils.scala:87)
at org.apache.spark.sql.execution.datasources.orc.OrcUtils$$anonfun$readSchema$1.applyOrElse(OrcUtils.scala:84)
at scala.collection.TraversableOnce$class.collectFirst(TraversableOnce.scala:145)
at scala.collection.AbstractIterator.collectFirst(Iterator.scala:1334)
at org.apache.spark.sql.execution.datasources.orc.OrcUtils$.readSchema(OrcUtils.scala:84)
at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.inferSchema(OrcFileFormat.scala:97)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:180)
at scala.Option.orElse(Option.scala:289)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:179)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:373)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.orc(DataFrameReader.scala:666)
at org.apache.spark.sql.DataFrameReader.orc(DataFrameReader.scala:656)
... 46 elided
Caused by: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not deserialize ATN with UUID 59627784-3be5-417a-b9eb-8131a7286089 (expected aadb8d7e-aeef-4415-ad2b-8204d6cf042e or a legacy UUID).
... 65 more
```

# Notes
There are other issues related to mixing ANT versions in the classloader from different kernels. If this is the case, which docker image I must to use (the one using the same ANT version that Spark)?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.