Azure / Azure/MachineLearningNotebooks

AzureML needs read-write permissions on its current working directory

Ouverte
#1,709 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Jupyter Notebook
Étoiles
4.4k
Forks
2.6k
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

Azure functions running on consumption, Elastic Premium and App Service Plans would leave the /home/site/**wwwroot** as ready only, creating Out Of Memory errors. But in reality the error is created due to a Read-Only
filesystem exception.

- A workaround to the problem is to use a custom container, since that would leave the wwwroot writable.

![Read-Only Erro](https://i.imgur.com/FVgN0H3.png)
`Result: Failure Exception: MemoryError: Engine process terminated. This is most likely due to system running out of memory. Please retry with increased memory. |session_id=l_c23cf85a-4ea0-424d-826c-a5105e38dfe4 Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 402, in _handle__invocation_request call_result = await self._loop.run_in_executor( File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 611, in _run_sync_func return ExtensionManager.get_sync_invocation_wrapper(context, File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper result = function(**args) File "/home/site/wwwroot/HttpTrigger/__init__.py", line 59, in main dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, '/home/train-dataset/tabular/iris.csv')]) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/_loggerfactory.py", line 132, in wrapper return func(*args, **kwargs) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/dataset_factory.py", line 357, in from_delimited_files dataflow = dataprep().read_csv(path, File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/readers.py", line 100, in read_csv df = Dataflow._path_to_get_files_block(path, archive_options) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/dataflow.py", line 2376, in _path_to_get_files_block return datastore_to_dataflow(path) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 40, in datastore_to_dataflow datastore, datastore_value = get_datastore_value(source) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 92, in get_datastore_value _set_auth_type(workspace) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 184, in _set_auth_type get_engine_api().set_aml_auth(SetAmlAuthMessageArgument(auth_type, json.dumps(auth_value))) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 19, in get_engine_api _engine_api = EngineAPI() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 103, in __init__ connect_to_requests_channel() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 99, in connect_to_requests_channel self._engine_server_secret = self.sync_host_secret(self.requests_channel.host_secret) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_aml_helper.py", line 38, in wrapper return send_message_func(op_code, message, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 304, in sync_host_secret response = self._message_channel.send_message('Engine.SyncHostSecret', message_args, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 275, in send_message raise message['error'] File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 223, in process_responses response = self._read_response(caller='MultiThreadMessageChannel.process_responses') File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 148, in _read_response raise error`

- Is there a way to point the SDK to write somewhere else in the container for example on the /home folder that is not set as read-only?


Function to repro the behavior:
[https://github.com/RoweKevin/AzureML-OOM-Repo](https://github.com/RoweKevin/AzureML-OOM-Repo)

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez par la reproduction liée de AzureML-OOM-Repo et le traceback à la ligne 59 de /home/site/wwwroot/HttpTrigger/__init__.py. Suivez le chemin du SDK via azureml/dataprep/api/engineapi/api.py et _datastore_helper.py, puis comparez le comportement entre les plans d’hébergement Azure listés. C’est terminé lorsque la reproduction n’échoue plus parce que le répertoire de travail actuel est en lecture seule, ou que l’emplacement alternatif pris en charge est clairement établi.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
azure, python
Domaine
cloud, machine-learning
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.