Azure / Azure/azureml-examples

MLTable - AzureML - Cache Environment variables

Open
#3,143 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Operating System

Linux

### Version Information

mltable-1.6.1
azureml-dataprep-rslex~=2.22.2dev0

### Steps to reproduce

1. Run a job on a compute that is size S
2. Mount Datastore as folder with mltables - **Datastore total size> S**
3. Wait...
4. Crash

For example, in AzureMachine Learning :
```python
storage_paths = [
{'folder':'azureml://subscriptions/$sub/resourcegroups/$rg/workspaces/$ws/datastores/$ds/paths/'}
]
tbl = mltable.from_paths(storage_paths )
mount_context = tbl._mount()
mount_context.start()
# Iterate over files

```
In order to fix my issue, i need to add extra mount settings :
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-read-write-data-v2?view=azureml-api-2&tabs=python#available-mount-settings

I use a wrapper class in order to do this on multiple storage / containers :
```python
@dataclass
class MyStorage:
mount_paths:List[int] = field(init=False,default_factory=list)
_is_mounted:bool = field(init=False,default=False)
_mount_context:Any = field(init=False,default=None)

def __post_init__(self):
os.environ['DATASET_MOUNT_CACHE_SIZE']="-40GB" # We leave at least 50GB available on the cluster.
os.environ['DATASET_MOUNT_BLOCK_BASED_CACHE_ENABLED']="True"

def mount(self):
print(f'Start Mounting storage...')
[print(f"- {path['folder']}") for path in self.mount_path]
tbl = mltable.from_paths(self.mount_paths)
self._mount_context = tbl._mount()
self._mount_context.start()
self._is_mounted = True
print(f'Mount Done - {self._mount_context.mount_point}')

def umount(self):
if self._is_mounted:
print(f'Start UnMounting - {self._mount_context.mount_point}')
self._mount_context.stop()
self._mount_context=None
self._is_mounted = False
print('UnMount Done...')

def __del__(self):
self.umount()

storage = MyStorage()
storage.mount_paths = storage_paths
storage.mount()
# Do stuff
del storage
```
I also tried to add the environment variable in the yaml job :
```yaml
$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json

experiment_name: LARGE-JOB
display_name: Large Job

environment_variables:
DATASET_MOUNT_CACHE_SIZE: "-40 GB"
DATASET_MOUNT_BLOCK_BASED_CACHE_ENABLED: "True"
DATASET_MOUNT_FILE_CACHE_PRUNE_TARGET: "0.0"

....

```
But none of theses solutions are working well.

### Expected behavior

I expect that the disk cache is pruned when it is reaching the -40GB limit on the compute machine.

### Actual behavior

Currently, the cache continues to grow :
![image](https://github.com/Azure/azureml-examples/assets/26071804/4b4487e8-01f1-4f3d-a958-d1a4e9131de3)

Until fail :
![image](https://github.com/Azure/azureml-examples/assets/26071804/09ae70d6-4751-4e20-b981-4c0cc76dd30a)

Even if i set environment variables in yaml :
![image](https://github.com/Azure/azureml-examples/assets/26071804/95aeb196-6115-4d37-b4b0-53f81f75d473)

or in code :
![image](https://github.com/Azure/azureml-examples/assets/26071804/a062ceda-6a42-4301-b626-1f80890436c6)

And i can confirm that the environment variable are used in the job :
![image](https://github.com/Azure/azureml-examples/assets/26071804/5fb9c425-bed7-48ae-ac13-f6057e299f1f)

**But it seems mltables are ignoring them.**

### Addition information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.