Azure / Azure/MachineLearningNotebooks

OutputFileDataset requires libfuse as dependency?

Open
#1,203 7 comments 0 reactions 0 assignees View on GitHub
ADO Data4ML product-issue
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

I recently upgraded my azureml-sdk to 1.16.0 and was running my PythonScriptStep roll.py spark dataset. But kept getting errors about libfuse.

- Region: westus2
- Run Id: `4ef30c7f-cf30-48a3-9d0c-a0e6882958ef`
- PipelineRun Id: `0e36abc1-afe3-4e56-b316-d8617d65149c`

`70_driver_log.txt`

```python
Logging warning in history service: ERROR:: Dataset failed. . Exception Details:Traceback (most recent call last):
File "/mnt/batch/tasks/shared/LS_root/jobs/avadevitsmlsvc/azureml/4ef30c7f-cf30-48a3-9d0c-a0e6882958ef/mounts/workspaceblobstore/azureml/4ef30c7f-cf30-48a3-9d0c-a0e6882958ef/azureml-setup/context_managers.py", line 385, in __enter__
self.datasets.__enter__()
File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/data/context_managers.py", line 108, in __enter__
self._mount_or_download(key, data_configuration)
File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/data/context_managers.py", line 175, in _mount_or_download
self._mount_readonly(name, dataset, target_path)
File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/data/context_managers.py", line 193, in _mount_readonly
mount_options = dataprep_fuse().MountOptions(free_space_required=free_space_required)
File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/data/_dataprep_helper.py", line 46, in dataprep_fuse
import azureml.dataprep.fuse.dprepfuse as _dprep_fuse
File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/dataprep/fuse/dprepfuse.py", line 4, in
from ._filecache import FileCache
File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/dataprep/fuse/_filecache.py", line 9, in
from .vendor.fuse import FuseOSError
File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/dataprep/fuse/vendor/fuse.py", line 115, in
raise EnvironmentError('Unable to find libfuse')
OSError: Unable to find libfuse
```

Here is my spark-requirement.txt file:
```azureml-sdk==1.16.0
azureml-dataprep[blobfuse]==2.4.0
fusepy==3.0.1
pyspark==2.4.4
pandas==0.25.3
pip==19.3.1
numpy==1.18.1
```
[packages in my local environment](https://gist.github.com/swanderz/c19bf0726fb2705988f43d30e368f5ba)

And here is a snippet of my pipeline with the spark config and PythonScriptStep:
```python
# Environment set up for PySpark Compute
spark_env = plf.get_environment(env_name='spark_env',
req_path=os.path.join(os.getcwd(), 'compute/aml_config/spark-requirements.txt'),
enable_docker=True,
docker_base_image='microsoft/mmlspark:0.16',
url=index_url + ' ' + mlpackage_req)

# use pyspark framework
spark_run_config = RunConfiguration(framework="pyspark")
spark_run_config.environment = spark_env

roll_step = PythonScriptStep(
name='roll.py',
script_name='roll.py',
arguments=['--input_dir', joined_data,
'--output_dir', rolled_data,
'--script_dir', ".",
'--min_date', '2015-06-30',
'--pct_rank', 'True'],
compute_target=compute_target_spark,
inputs=[joined_data],
outputs=[rolled_data],
runconfig=spark_run_config,
source_directory=os.path.join(os.getcwd(), 'compute', 'roll'),
allow_reuse=pipeline_reuse
)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.