dask / dask/dask-cloudprovider
Azure : RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'LZ4', 'UNCOMPRESSED']
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
### Steps to reproduce:
I have created Dask Cluster inside AzureML environment using the following code:
```python
amlcluster = AzureMLCluster(ws,
vm_size="STANDARD_D1",
environment_definition=ws.environments['AzureML-Dask-CPU'],
initial_node_count=0,
scheduler_idle_timeout=10800,
vnet='vnet',
subnet='subnet',
vnet_resource_group='resourcegroup',
ct_name="biswasdask",
)
```
Next open the jupyter lab using the link returned by `amlcluster.jupyter_link`
As per my understanding I am into the scheduler node of the cluster now.
On the Jupyter notebook, try the following code (from the repository [azureml-examples](https://github.com/Azure/azureml-examples/blob/main/tutorials/using-dask/1.intro-to-dask.ipynb)):
```python
from adlfs import AzureBlobFileSystem
container_name = "isdweatherdatacontainer"
storage_options = {"account_name": "azureopendatastorage"}
fs = AzureBlobFileSystem(**storage_options)
files = fs.glob(f"{container_name}/ISDWeather/year=2020/month=2/part-00003-tid-695161346761253622-368439cf-81e6-43f1-be5d-49ba29e282c0-2567-2.c000.snappy.parquet")
ddf = dd.read_parquet(files, storage_options=storage_options, chunksize="20MB")
ddf.head()
```
It returns the following error:
`RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'LZ4', 'UNCOMPRESSED']`
This is seems to be an old [issue](https://github.com/dask/fastparquet/issues/366). But, since I have not created this environment manually, I don't know what is the problem?
Contributor guide
Assessment
This issue has not been assessed yet.