dask / dask/dask-cloudprovider

Azure : RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'LZ4', 'UNCOMPRESSED']

Open
#177 3 comments 0 reactions 0 assignees View on GitHub
provider/azure/azureml
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

### Steps to reproduce:

I have created Dask Cluster inside AzureML environment using the following code:

```python
amlcluster = AzureMLCluster(ws,
vm_size="STANDARD_D1",
environment_definition=ws.environments['AzureML-Dask-CPU'],
initial_node_count=0,
scheduler_idle_timeout=10800,
vnet='vnet',
subnet='subnet',
vnet_resource_group='resourcegroup',
ct_name="biswasdask",
)
```

Next open the jupyter lab using the link returned by `amlcluster.jupyter_link`

As per my understanding I am into the scheduler node of the cluster now.

On the Jupyter notebook, try the following code (from the repository [azureml-examples](https://github.com/Azure/azureml-examples/blob/main/tutorials/using-dask/1.intro-to-dask.ipynb)):

```python
from adlfs import AzureBlobFileSystem

container_name = "isdweatherdatacontainer"
storage_options = {"account_name": "azureopendatastorage"}

fs = AzureBlobFileSystem(**storage_options)
files = fs.glob(f"{container_name}/ISDWeather/year=2020/month=2/part-00003-tid-695161346761253622-368439cf-81e6-43f1-be5d-49ba29e282c0-2567-2.c000.snappy.parquet")
ddf = dd.read_parquet(files, storage_options=storage_options, chunksize="20MB")

ddf.head()
```
It returns the following error:

`RuntimeError: Decompression 'SNAPPY' not available. Options: ['GZIP', 'LZ4', 'UNCOMPRESSED']`

This is seems to be an old [issue](https://github.com/dask/fastparquet/issues/366). But, since I have not created this environment manually, I don't know what is the problem?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.