Azure / Azure/MachineLearningNotebooks

AzureML needs read-write permissions on its current working directory

未關閉
#1,709 2 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Jupyter Notebook
星號
4.4k
分支
2.6k
PR 合併指標
30 天內沒有已合併 PR

描述

Azure functions running on consumption, Elastic Premium and App Service Plans would leave the /home/site/**wwwroot** as ready only, creating Out Of Memory errors. But in reality the error is created due to a Read-Only
filesystem exception.

- A workaround to the problem is to use a custom container, since that would leave the wwwroot writable.

![Read-Only Erro](https://i.imgur.com/FVgN0H3.png)
`Result: Failure Exception: MemoryError: Engine process terminated. This is most likely due to system running out of memory. Please retry with increased memory. |session_id=l_c23cf85a-4ea0-424d-826c-a5105e38dfe4 Stack: File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 402, in _handle__invocation_request call_result = await self._loop.run_in_executor( File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 611, in _run_sync_func return ExtensionManager.get_sync_invocation_wrapper(context, File "/azure-functions-host/workers/python/3.8/LINUX/X64/azure_functions_worker/extension.py", line 215, in _raw_invocation_wrapper result = function(**args) File "/home/site/wwwroot/HttpTrigger/__init__.py", line 59, in main dataset = Dataset.Tabular.from_delimited_files(path = [(datastore, '/home/train-dataset/tabular/iris.csv')]) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/_loggerfactory.py", line 132, in wrapper return func(*args, **kwargs) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/data/dataset_factory.py", line 357, in from_delimited_files dataflow = dataprep().read_csv(path, File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/readers.py", line 100, in read_csv df = Dataflow._path_to_get_files_block(path, archive_options) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/dataflow.py", line 2376, in _path_to_get_files_block return datastore_to_dataflow(path) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 40, in datastore_to_dataflow datastore, datastore_value = get_datastore_value(source) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 92, in get_datastore_value _set_auth_type(workspace) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_datastore_helper.py", line 184, in _set_auth_type get_engine_api().set_aml_auth(SetAmlAuthMessageArgument(auth_type, json.dumps(auth_value))) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 19, in get_engine_api _engine_api = EngineAPI() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 103, in __init__ connect_to_requests_channel() File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 99, in connect_to_requests_channel self._engine_server_secret = self.sync_host_secret(self.requests_channel.host_secret) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/_aml_helper.py", line 38, in wrapper return send_message_func(op_code, message, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/api.py", line 304, in sync_host_secret response = self._message_channel.send_message('Engine.SyncHostSecret', message_args, cancellation_token) File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 275, in send_message raise message['error'] File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 223, in process_responses response = self._read_response(caller='MultiThreadMessageChannel.process_responses') File "/home/site/wwwroot/.python_packages/lib/site-packages/azureml/dataprep/api/engineapi/engine.py", line 148, in _read_response raise error`

- Is there a way to point the SDK to write somewhere else in the container for example on the /home folder that is not set as read-only?


Function to repro the behavior:
[https://github.com/RoweKevin/AzureML-OOM-Repo](https://github.com/RoweKevin/AzureML-OOM-Repo)

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

從連結的 AzureML-OOM-Repo 重現案例以及 /home/site/wwwroot/HttpTrigger/__init__.py 第 59 行的 traceback 開始。沿著 SDK 路徑檢查 azureml/dataprep/api/engineapi/api.py 和 _datastore_helper.py,然後比較所列 Azure 託管方案之間的行為。完成的標準是:重現不再因目前工作目錄為唯讀而失敗,或明確確定受支援的替代位置。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
azure, python
領域
cloud, machine-learning
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。