twosigma / twosigma/flint

java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.internalCreateDataFrame(Lorg

Open
#26 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
1.2k
Forks
199
PR merge metrics
No merged PRs in 30d

Description

Hi - I am getting the following error when trying to run the Python example:
```
Traceback (most recent call last):
File "", line 1, in
File "/Users/__/flint-master/python/ts/flint/dataframe.py", line 592, in summarize
tsrdd = self.timeSeriesRDD.summarize(composed_summarizer._jsummarizer(self._sc), scala_key)
File "/Users/__/flint-master/python/ts/flint/dataframe.py", line 133, in timeSeriesRDD
self._jdf, self._is_sorted, self._junit, self._time_column)
File "/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1160, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/Users/__/spark-2.3.0-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/local/lib/python3.6/site-packages/py4j/protocol.py", line 320, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o284.fromDF.
java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.internalCreateDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;
at org.apache.spark.sql.DFConverter$.toDataFrame(DFConverter.scala:37)
at com.twosigma.flint.timeseries.TimeSeriesStore$.apply(TimeSeriesStore.scala:72)
at com.twosigma.flint.timeseries.TimeSeriesStore$.apply(TimeSeriesStore.scala:59)
at com.twosigma.flint.timeseries.TimeSeriesRDD$.fromDFWithPartInfo(TimeSeriesRDD.scala:388)
at com.twosigma.flint.timeseries.TimeSeriesRDD$.fromDF(TimeSeriesRDD.scala:271)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
```
Is this due to Spark support being limited to 2.0?
**Thank you!!**

Contributor guide

Open the contributing guide

Research direction

Start with the failing Python example and compare the Spark 2.3.0 environment against the Scala call sites named in the trace: DFConverter.scala, TimeSeriesStore.scala, and TimeSeriesRDD.scala. Verify which Spark versions those entry points support and reproduce the NoSuchMethodError. Done means the supported version or compatibility problem is established and documented in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, scala
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.