[Java] PySpark3 with pandas 1.1.5 and pyarrow 2.0.0 getting the below error
- 主要語言
- Java
- 星號
- 94
- 分支
- 152
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 11
描述
While running pyspark3 with pandas 1.1.5 and pyarrow 2.0.0 getting the below error:
**Spark Code:**
```java
import pyarrow
import pandas as pd
df = pd.DataFrame({'col1' : [1,2,3], 'col2': [4,5,6]})
df_sp = spark.createDataFrame(df)
df_sp.cache().count()
schema = df_sp.schema
def dummy_udf(data):
return data
res = df_sp.groupby('col1').applyInPandas(dummy_udf, schema=schema)
print(res.cache().count())
print(res.toPandas())
```
**Exception:**
```java
21/09/17 07:28:10 ERROR util.Utils: Uncaught exception in thread stdout writer for python3
java.lang.NoSuchMethodError: com.google.flatbuffers.FlatBufferBuilder.createString(Ljava/lang/CharSequence;)I
at org.apache.arrow.vector.types.pojo.Field.getField(Field.java:204)
at org.apache.arrow.vector.types.pojo.Schema.getSchema(Schema.java:178)
at org.apache.arrow.vector.ipc.message.MessageSerializer.serializeMetadata(MessageSerializer.java:187)
at org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:165)
at org.apache.arrow.vector.ipc.ArrowWriter.ensureStarted(ArrowWriter.java:159)
at org.apache.arrow.vector.ipc.ArrowWriter.start(ArrowWriter.java:112)
at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.$anonfun$writeIteratorToStream$1(ArrowPythonRunner.scala:86)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.writeIteratorToStream(ArrowPythonRunner.scala:103)
at org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(PythonRunner.scala:397)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996)
at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:232)
21/09/17 07:28:10 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[stdout writer for python3,5,main]
java.lang.NoSuchMethodError: com.google.flatbuffers.FlatBufferBuilder.createString(Ljava/lang/CharSequence;)I
at org.apache.arrow.vector.types.pojo.Field.getField(Field.java:204)
at org.apache.arrow.vector.types.pojo.Schema.getSchema(Schema.java:178)
at org.apache.arrow.vector.ipc.message.MessageSerializer.serializeMetadata(MessageSerializer.java:187)
at org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(MessageSerializer.java:165)
at org.apache.arrow.vector.ipc.ArrowWriter.ensureStarted(ArrowWriter.java:159)
at org.apache.arrow.vector.ipc.ArrowWriter.start(ArrowWriter.java:112)
at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.$anonfun$writeIteratorToStream$1(ArrowPythonRunner.scala:86)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.writeIteratorToStream(ArrowPythonRunner.scala:103)
at org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(PythonRunner.scala:397)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996)
at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:232)
21/09/17 07:28:10 WARN storage.BlockManager: Putting block rdd_25_69 failed due to exception org.apache.spark.SparkException: Python worker exited unexpectedly (crashed).
21/09/17 07:28:10 INFO memory.MemoryStore: MemoryStore cleared
21/09/17 07:28:10 INFO storage.BlockManager: BlockManager stopped
21/09/17 07:28:10 INFO util.ShutdownHookManager: Shutdown hook called
```
**Reporter**: [Ranga Reddy](https://issues.apache.org/jira/browse/ARROW-14038)
**Note**: *This issue was originally created as [ARROW-14038](https://issues.apache.org/jira/browse/ARROW-14038). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
貢獻指南
研究方向
使用 pandas 1.1.5 和 pyarrow 2.0.0 重現所提供的 PySpark 範例,然後從 ArrowPythonRunner 經過 ArrowWriter 追蹤故障,直到 org.apache.arrow.vector.types.pojo.Field.getField。將 createString 中涉及的 Java FlatBuffers 相依性與 issue 中連結的遷移文件進行比較;當相容性問題和適用的遷移路徑已記錄或驗證後,即視為完成。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- java, pandas, python
- 領域
- backend, distributed-systems
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 25/100