Azure / Azure/MachineLearningNotebooks

Register pandas dataframe memory issues

Đang mở
#1,585 5 bình luận 0 reaction 0 người được giao Xem trên GitHub
ADO Data4ML product-issue
Ngôn ngữ chính
Jupyter Notebook
Star
4.4k
Fork
2.6k
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

When using the `register_pandas_dataframe()` method as suggested by the tip [here](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-register-datasets#create-a-dataset-from-pandas-dataframe), I get a system memory error:
```
Error Code: ScriptExecution.ReadDataFrame.Unexpected
Failed Step: 1d7c20d4-6b70-4075-8b68-74f51264bf5b
Error Message: ScriptExecutionException was caused by ReadDataFrameException. Unexpected exception during ReadDataframeFromSocket.
Failed to read DataFrame from host. Exception of type 'System.OutOfMemoryException' was thrown.
```

When I run the same method with the same dataframe from a jupyter notebook it works as expected. I have ensured I have enough memory on my system when running the script (1.42 MiB dataframe and 6GB RAM free) so I don't think that's an issue. I know this method is experimental, so I am using the older method and it's working fine.

Another issue that sometimes happens (if it's not the system memory issue) is a streamAccessValidation error
```
azureml.dataprep.api.errorhandlers.ExecutionError:
Error Code: ScriptExecution.ReadDataFrame.StreamAccess.Validation
Validation Error Code: Invalid
Validation Target: PreppyFile
Failed Step: bfd9a1d5-c01f-485a-8761-99cd0a41d0c3
Error Message: ScriptExecutionException was caused by ReadDataFrameException.
Failed to read Pandas DataFrame form Python host. Make sure Dataflow is created directly from the source Pandas DataFrame.
StreamAccessException was caused by ValidationException.
Trying to read an invalid file. Missing sentinel value in the beginning
| session_id=710cc9c3-4478-4be9-998c-0e4a009800f5
```
Again, this does not happen when using the method from a jupyter notebook, only when running a script on my local machine.

---
#### Document Details

⚠ *Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.*

* ID: 9b1aee32-5a1f-2ef1-ee76-30b17999db87
* Version Independent ID: f4785272-a81d-e9cc-c6a5-9d41c8dcd90b
* Content: [azureml.data.dataset_factory.TabularDatasetFactory class - Azure Machine Learning Python](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.dataset_factory.tabulardatasetfactory?view=azure-ml-py#azureml_data_dataset_factory_TabularDatasetFactory_register_pandas_dataframe)
* Content Source: [AzureML-Docset/stable/docs-ref-autogen/azureml-core/azureml.data.dataset_factory.TabularDatasetFactory.yml](https://github.com/MicrosoftDocs/MachineLearning-Python-pr/blob/live/AzureML-Docset/stable/docs-ref-autogen/azureml-core/azureml.data.dataset_factory.TabularDatasetFactory.yml)
* Service: **machine-learning**
* Sub-service: **core**
* GitHub Login: @DebFro
* Microsoft Alias: **debfro**

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.