Azure / Azure/MachineLearningNotebooks

AzureML needs read-write permissions on its current working directory

Aperta
#1,709 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Jupyter Notebook
Stelle
4.4k
Fork
2.6k
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Azure functions running on consumption, Elastic Premium and App Service Plans would leave the /home/site/**wwwroot** as ready only, creating Out Of Memory errors. But in reality the error is created due to a Read-Only
filesystem exception.

- A workaround to the problem is to use a custom container, since that would leave the wwwroot writable.

![Read-Only Erro](https://i.imgur.com/FVgN0H3.png)
`Result: Failure Exception: MemoryError: Engine process terminated. This is most likely due to system running out of memory. Please retry with increased memory. |session_id=l_c23cf85a-4ea0-424d-826c-a5105e38dfe4 Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 402, in _handle__invocation_request call_result = await self._loop.run_in_executor( File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 611, in _run_sync_func return ExtensionManager.get_sync_invocation_wrapper(context, File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper result = function(**args) File "/home/site/wwwroot/HttpTrigger/__init__.py", line 59, in main dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, '/home/train-dataset/tabular/iris.csv')]) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/_loggerfactory.py", line 132, in wrapper return func(*args, **kwargs) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/dataset_factory.py", line 357, in from_delimited_files dataflow = dataprep().read_csv(path, File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/readers.py", line 100, in read_csv df = Dataflow._path_to_get_files_block(path, archive_options) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/dataflow.py", line 2376, in _path_to_get_files_block return datastore_to_dataflow(path) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 40, in datastore_to_dataflow datastore, datastore_value = get_datastore_value(source) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 92, in get_datastore_value _set_auth_type(workspace) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 184, in _set_auth_type get_engine_api().set_aml_auth(SetAmlAuthMessageArgument(auth_type, json.dumps(auth_value))) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 19, in get_engine_api _engine_api = EngineAPI() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 103, in __init__ connect_to_requests_channel() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 99, in connect_to_requests_channel self._engine_server_secret = self.sync_host_secret(self.requests_channel.host_secret) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_aml_helper.py", line 38, in wrapper return send_message_func(op_code, message, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 304, in sync_host_secret response = self._message_channel.send_message('Engine.SyncHostSecret', message_args, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 275, in send_message raise message['error'] File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 223, in process_responses response = self._read_response(caller='MultiThreadMessageChannel.process_responses') File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 148, in _read_response raise error`

- Is there a way to point the SDK to write somewhere else in the container for example on the /home folder that is not set as read-only?


Function to repro the behavior:
[https://github.com/RoweKevin/AzureML-OOM-Repo](https://github.com/RoweKevin/AzureML-OOM-Repo)

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.