Azure / Azure/MachineLearningNotebooks

Run.complete() fails if outputs dir contains '..' in path

未關閉
#1,500 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
ADO bug SDK
主要語言
Jupyter Notebook
星號
4.4k
分支
2.6k
PR 合併指標
30 天內沒有已合併 PR

描述

The SDK should use `os.abspath()` ([stack overflow answer](https://stackoverflow.com/a/32838876/3842610)) before trying to upload the files.

[Reproducible example](https://github.com/swanderz/MachineLearningNotebooks/blob/19d2c7530ad6b07dcb6ff944c75135849abc4332/how-to-use-azureml/track-and-monitor-experiments/logging-api/logging-api.ipynb)

```python
from azureml.core import Experiment, Workspace, Run
import os
ws = Workspace.from_config()
experiment = Experiment(workspace=ws, name='logging-api-test')

# make a folder using the double dot to indicate parent folder
directory = '../logging-api'

if not os.path.exists(directory):
os.mkdir(directory)

# start a run
run = experiment.start_logging(outputs=directory)

# write a file to watched output folder
file_name = os.path.join(directory, "myfile.txt")
with open(file_name, "w") as f:
f.write('This is an output file that will be uploaded.\n')

# this will fail!
run2.complete()
```

stack trace
```
AzureMLAggregatedException Traceback (most recent call last)
in
----> 1 run3.complete()

~/opt/miniconda3/envs/rvo/lib/python3.8/site-packages/azureml/core/run.py in complete(self, _set_status)
1341 :type _set_status: bool
1342 """
-> 1343 self._client.complete(_set_status=_set_status)
1344
1345 def fail(self, error_details=None, error_code=None, _set_status=True):

~/opt/miniconda3/envs/rvo/lib/python3.8/site-packages/azureml/_run_impl/run_history_facade.py in complete(self, _set_status)
592 """
593 self.flush()
--> 594 self.upload_tracked_files()
595 self.set_completed_status(_set_status=_set_status)
596

~/opt/miniconda3/envs/rvo/lib/python3.8/site-packages/azureml/_run_impl/run_history_facade.py in upload_tracked_files(self)
658
659 def upload_tracked_files(self):
--> 660 self.output_file_tracker.upload_tracked_files()
661
662 def take_snapshot(self, file_or_folder_path, _raise_on_validation_failure=True):

~/opt/miniconda3/envs/rvo/lib/python3.8/site-packages/azureml/_history/utils/context_managers.py in upload_tracked_files(self)
151
152 def upload_tracked_files(self):
--> 153 self.py_wd.track(self.artifacts_client, self.container_id, self.trackfolders, self.deny_list)
154
155

~/opt/miniconda3/envs/rvo/lib/python3.8/site-packages/azureml/_history/utils/context_managers.py in track(self, artifacts_client, container_id, track_folders, blacklist)
93 for fs in self.fs_list:
94 self.logger.debug("Calling track for {}".format(fs.ident()))
---> 95 fs.track(artifacts_client, container_id, track_folders, blacklist)
96 return True
97

~/opt/miniconda3/envs/rvo/lib/python3.8/site-packages/azureml/_history/utils/filesystem.py in track(self, artifacts_client, container_id, track_folders, blacklist)
79
80 if len(exception_messages) > 0:
---> 81 raise AzureMLAggregatedException(exception_messages)
82
83 def _upload_folder(self, artifacts_client, container_id, path, blacklist):

AzureMLAggregatedException: AzureMLAggregatedException:
Message: Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.
StorageAccount: sasetfdevdsresvalue
ContainerName: azureml
StatusCode: 403
InnerException None
ErrorResponse
{
"error": {
"message": "Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.\n\tStorageAccount: sasetfdevdsresvalue\n\tContainerName: azureml\n\tStatusCode: 403"
}
}
```

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

從 issue 中連結的可重現 logging-api notebook 開始,追蹤 Run.complete() 經過 upload_tracked_files() 的執行流程。使用包含 '..' 的 outputs 目錄重現該失敗;當被追蹤的檔案成功上傳,且完成操作不再回報授權錯誤時,即表示完成。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
azure, jupyter-notebook, python
領域
cloud, machine-learning
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。