Azure / Azure/MachineLearningNotebooks
AzureML needs read-write permissions on its current working directory
- Lingua principale
- Jupyter Notebook
- Stelle
- 4.4k
- Fork
- 2.6k
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
Azure functions running on consumption, Elastic Premium and App Service Plans would leave the /home/site/**wwwroot** as ready only, creating Out Of Memory errors. But in reality the error is created due to a Read-Only
filesystem exception.
- A workaround to the problem is to use a custom container, since that would leave the wwwroot writable.

`Result: Failure Exception: MemoryError: Engine process terminated. This is most likely due to system running out of memory. Please retry with increased memory. |session_id=l_c23cf85a-4ea0-424d-826c-a5105e38dfe4 Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 402, in _handle__invocation_request call_result = await self._loop.run_in_executor( File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 611, in _run_sync_func return ExtensionManager.get_sync_invocation_wrapper(context, File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper result = function(**args) File "/home/site/wwwroot/HttpTrigger/__init__.py", line 59, in main dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, '/home/train-dataset/tabular/iris.csv')]) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/_loggerfactory.py", line 132, in wrapper return func(*args, **kwargs) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/dataset_factory.py", line 357, in from_delimited_files dataflow = dataprep().read_csv(path, File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/readers.py", line 100, in read_csv df = Dataflow._path_to_get_files_block(path, archive_options) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/dataflow.py", line 2376, in _path_to_get_files_block return datastore_to_dataflow(path) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 40, in datastore_to_dataflow datastore, datastore_value = get_datastore_value(source) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 92, in get_datastore_value _set_auth_type(workspace) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 184, in _set_auth_type get_engine_api().set_aml_auth(SetAmlAuthMessageArgument(auth_type, json.dumps(auth_value))) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 19, in get_engine_api _engine_api = EngineAPI() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 103, in __init__ connect_to_requests_channel() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 99, in connect_to_requests_channel self._engine_server_secret = self.sync_host_secret(self.requests_channel.host_secret) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_aml_helper.py", line 38, in wrapper return send_message_func(op_code, message, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 304, in sync_host_secret response = self._message_channel.send_message('Engine.SyncHostSecret', message_args, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 275, in send_message raise message['error'] File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 223, in process_responses response = self._read_response(caller='MultiThreadMessageChannel.process_responses') File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 148, in _read_response raise error`
- Is there a way to point the SDK to write somewhere else in the container for example on the /home folder that is not set as read-only?
Function to repro the behavior:
[https://github.com/RoweKevin/AzureML-OOM-Repo](https://github.com/RoweKevin/AzureML-OOM-Repo)
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.