java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.internalCreateDataFrame(Lorg
- Dominant language
- Scala
- Stars
- 1.2k
- Forks
- 199
- PR merge metrics
- No merged PRs in 30d
Description
Hi - I am getting the following error when trying to run the Python example:
```
Traceback (most recent call last):
File "", line 1, in
File "/Users/__/flint-master/python/ts/flint/dataframe.py", line 592, in summarize
tsrdd = self.timeSeriesRDD.summarize(composed_summarizer._jsummarizer(self._sc), scala_key)
File "/Users/__/flint-master/python/ts/flint/dataframe.py", line 133, in timeSeriesRDD
self._jdf, self._is_sorted, self._junit, self._time_column)
File "/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py", line 1160, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/Users/__/spark-2.3.0-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/local/lib/python3.6/site-packages/py4j/protocol.py", line 320, in get_return_value
format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o284.fromDF.
java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.internalCreateDataFrame(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/Dataset;
at org.apache.spark.sql.DFConverter$.toDataFrame(DFConverter.scala:37)
at com.twosigma.flint.timeseries.TimeSeriesStore$.apply(TimeSeriesStore.scala:72)
at com.twosigma.flint.timeseries.TimeSeriesStore$.apply(TimeSeriesStore.scala:59)
at com.twosigma.flint.timeseries.TimeSeriesRDD$.fromDFWithPartInfo(TimeSeriesRDD.scala:388)
at com.twosigma.flint.timeseries.TimeSeriesRDD$.fromDF(TimeSeriesRDD.scala:271)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
```
Is this due to Spark support being limited to 2.0?
**Thank you!!**
Contributor guide
Research direction
Start with the failing Python example and compare the Spark 2.3.0 environment against the Scala call sites named in the trace: DFConverter.scala, TimeSeriesStore.scala, and TimeSeriesRDD.scala. Verify which Spark versions those entry points support and reproduce the NoSuchMethodError. Done means the supported version or compatibility problem is established and documented in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100