Azure / Azure/MachineLearningNotebooks
Register pandas dataframe memory issues
- Lenguaje dominante
- Jupyter Notebook
- Estrellas
- 4.4k
- Forks
- 2.6k
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
When using the `register_pandas_dataframe()` method as suggested by the tip [here](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-register-datasets#create-a-dataset-from-pandas-dataframe), I get a system memory error:
```
Error Code: ScriptExecution.ReadDataFrame.Unexpected
Failed Step: 1d7c20d4-6b70-4075-8b68-74f51264bf5b
Error Message: ScriptExecutionException was caused by ReadDataFrameException. Unexpected exception during ReadDataframeFromSocket.
Failed to read DataFrame from host. Exception of type 'System.OutOfMemoryException' was thrown.
```
When I run the same method with the same dataframe from a jupyter notebook it works as expected. I have ensured I have enough memory on my system when running the script (1.42 MiB dataframe and 6GB RAM free) so I don't think that's an issue. I know this method is experimental, so I am using the older method and it's working fine.
Another issue that sometimes happens (if it's not the system memory issue) is a streamAccessValidation error
```
azureml.dataprep.api.errorhandlers.ExecutionError:
Error Code: ScriptExecution.ReadDataFrame.StreamAccess.Validation
Validation Error Code: Invalid
Validation Target: PreppyFile
Failed Step: bfd9a1d5-c01f-485a-8761-99cd0a41d0c3
Error Message: ScriptExecutionException was caused by ReadDataFrameException.
Failed to read Pandas DataFrame form Python host. Make sure Dataflow is created directly from the source Pandas DataFrame.
StreamAccessException was caused by ValidationException.
Trying to read an invalid file. Missing sentinel value in the beginning
| session_id=710cc9c3-4478-4be9-998c-0e4a009800f5
```
Again, this does not happen when using the method from a jupyter notebook, only when running a script on my local machine.
---
#### Document Details
⚠ *Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.*
* ID: 9b1aee32-5a1f-2ef1-ee76-30b17999db87
* Version Independent ID: f4785272-a81d-e9cc-c6a5-9d41c8dcd90b
* Content: [azureml.data.dataset_factory.TabularDatasetFactory class - Azure Machine Learning Python](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.dataset_factory.tabulardatasetfactory?view=azure-ml-py#azureml_data_dataset_factory_TabularDatasetFactory_register_pandas_dataframe)
* Content Source: [AzureML-Docset/stable/docs-ref-autogen/azureml-core/azureml.data.dataset_factory.TabularDatasetFactory.yml](https://github.com/MicrosoftDocs/MachineLearning-Python-pr/blob/live/AzureML-Docset/stable/docs-ref-autogen/azureml-core/azureml.data.dataset_factory.TabularDatasetFactory.yml)
* Service: **machine-learning**
* Sub-service: **core**
* GitHub Login: @DebFro
* Microsoft Alias: **debfro**
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Comience con el punto de entrada register_pandas_dataframe() y compare su comportamiento en el script local con el flujo de trabajo del cuaderno de Jupyter descrito en el informe. Reproduzca los errores OutOfMemoryException y StreamAccess.Validation y, después, siga la ruta de registro de datos de Azure Machine Learning; el trabajo estará terminado cuando se haya identificado una causa reproducible y se haya documentado o corregido el flujo de trabajo afectado.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- azure, pandas, python
- Área
- data, machine-learning
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 25/100