Azure / Azure/MachineLearningNotebooks
Permission denied (read-only file system)
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Description
### Situation
We are using AzureML pipelines to train neural networks.
The raw data used to train the model is saved on our Azure blob storage.
Just as the trained models (and the checkpoints while training) are saved here.
This has worked perfectly for the last few months, more than 50 pipelines were ran that way.
### Issue
Since a few days, we constantly get the same error when trying to train a model.
After the first epoch, where the model tries to save the current checkpoint, but permission is denied.
Permission seems to be denied because it's a read-only file system.
However, none of the access rights seem to imply that this file is any different.
All the other files (images, annotations) do not seem to raise any issue.
Even rerunning the exact same pipelines on the exact same data that worked perfectly last week, result in this error.
> tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at save_restore_v2_ops.cc:157 : Permission denied: /mnt/azureml/cr/j/0ddbaa5dfd4243c4bd18feabd6037209/cap/data-capability/wd/output_84f84eab_univision_ai/pc_ds_2021_v3/refinement-reg/v2/results/128_c1720fb5-81ed-45aa-a823-a1fa5ef1a8d1/export/saved_model/variables/variables_temp/part-00000-of-00001.data-00000-of-00001.tempstate5255274353572806690; Read-only file system
...
Epoch 00001: val_loss improved from inf to 0.10273, saving model to /mnt/azureml/cr/j/0ddbaa5dfd4243c4bd18feabd6037209/cap/data-capability/wd/output_84f84eab_univision_ai/pc_ds_2021_v3/refinement-reg/v2/results/128_c1720fb5-81ed-45aa-a823-a1fa5ef1a8d1/export/saved_model
Cleaning up all outstanding Run operations, waiting 300.0 seconds
2 items cleaning up...
Cleanup took 0.19626808166503906 seconds
...
tensorflow.python.framework.errors_impl.PermissionDeniedError: /mnt/azureml/cr/j/0ddbaa5dfd4243c4bd18feabd6037209/cap/data-capability/wd/output_84f84eab_univision_ai/pc_ds_2021_v3/refinement-reg/v2/results/128_c1720fb5-81ed-45aa-a823-a1fa5ef1a8d1/export/saved_model/variables/variables_temp/part-00000-of-00001.data-00000-of-00001.tempstate5255274353572806690; Read-only file system [Op:SaveV2]
What more can I try or check?
There were no package version changes, it all stayed the same.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.