Azure / Azure/MachineLearningNotebooks
Key not found "ADLSGen2" when using `to_spark_dataframe`
- 主要語言
- Jupyter Notebook
- 星號
- 4.4k
- 分支
- 2.6k
- PR 合併指標
- 30 天內沒有已合併 PR
描述
I'm creating a dataset directly using a URL (relying on identity-based access):
```python
dataset = Dataset.Tabular.from_parquet_files("https://.dfs.core.windows.net/")
```
(This prompts my browser to start a login-process.)
While `dataset.to_pandas_dataframe()` works fine, when I try `dataset.to_spark_dataframe()` I get the following Java traceback:
```
: java.util.NoSuchElementException: key not found: ADLSGen2
at scala.collection.MapLike.default(MapLike.scala:235)
at scala.collection.MapLike.default$(MapLike.scala:234)
at scala.collection.AbstractMap.default(Map.scala:63)
at scala.collection.MapLike.apply(MapLike.scala:144)
at scala.collection.MapLike.apply$(MapLike.scala:143)
at scala.collection.AbstractMap.apply(Map.scala:63)
at com.microsoft.dprep.io.StreamInfoFileSystem$.toFileSystemPath(StreamInfoFileSystem.scala:68)
at com.microsoft.dprep.execution.Storage$.expandHdfsPath(Storage.scala:37)
at com.microsoft.dprep.execution.executors.GetFilesExecutor$.$anonfun$getFiles$1(GetFilesExecutor.scala:18)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
at com.microsoft.dprep.execution.executors.GetFilesExecutor$.getFiles(GetFilesExecutor.scala:12)
at com.microsoft.dprep.execution.LariatDataset$.getFiles(LariatDataset.scala:32)
at com.microsoft.dprep.execution.PySparkExecutor.getFiles(PySparkExecutor.scala:225)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:834)
```
This is using "com.microsoft.ml.spark:mmlspark_2.12:1.0.0-rc3-62-25d40cff-SNAPSHOT" and PySpark 3.1.2.
What might cause this error?
The Java code is called from a generated Python module which shows where the "ADLSGen2" key comes from:
```python
# ...
lds0 = jex.getFiles(
[{"searchPattern":"https://.dfs.core.windows.net/",
"handler":"ADLSGen2",
"arguments":{"credential":""}
}],
secrets
)
# ...
```
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
從包含處理程序 ADLSGen2 的產生式 Python getFiles 呼叫以及 traceback 項目 StreamInfoFileSystem.toFileSystemPath 開始。追蹤該處理程序如何針對所述的 Spark 和 mmlspark 版本進行註冊。確認原因並讓 to_spark_dataframe 能夠使用基於身分的 URL 運作,即視為完成。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- azure, python, spark
- 領域
- cloud, data-engineering
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 需要釐清
- 新手友好度
- 20/100