OutputFileDataset requires libfuse as dependency?

I recently upgraded my azureml-sdk to 1.16.0 and was running my PythonScriptStep roll.py spark dataset. But kept getting errors about libfuse.

• Region: westus2
• Run Id: 4ef30c7f-cf30-48a3-9d0c-a0e6882958ef
• PipelineRun Id: 0e36abc1-afe3-4e56-b316-d8617d65149c

70_driver_log.txt

Logging warning in history service: ERROR:: Dataset  failed. . Exception Details:Traceback (most recent call last):
  File "/mnt/batch/tasks/shared/LS_root/jobs/avadevitsmlsvc/azureml/4ef30c7f-cf30-48a3-9d0c-a0e6882958ef/mounts/workspaceblobstore/azureml/4ef30c7f-cf30-48a3-9d0c-a0e6882958ef/azureml-setup/context_managers.py", line 385, in __enter__
    self.datasets.__enter__()
  File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/data/context_managers.py", line 108, in __enter__
    self._mount_or_download(key, data_configuration)
  File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/data/context_managers.py", line 175, in _mount_or_download
    self._mount_readonly(name, dataset, target_path)
  File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/data/context_managers.py", line 193, in _mount_readonly
    mount_options = dataprep_fuse().MountOptions(free_space_required=free_space_required)
  File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/data/_dataprep_helper.py", line 46, in dataprep_fuse
    import azureml.dataprep.fuse.dprepfuse as _dprep_fuse
  File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/dataprep/fuse/dprepfuse.py", line 4, in <module>
    from ._filecache import FileCache
  File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/dataprep/fuse/_filecache.py", line 9, in <module>
    from .vendor.fuse import FuseOSError
  File "/azureml-envs/azureml_29037f86d36e2b5b4f047e45f790fdb9/lib/python3.6/site-packages/azureml/dataprep/fuse/vendor/fuse.py", line 115, in <module>
    raise EnvironmentError('Unable to find libfuse')
OSError: Unable to find libfuse

Here is my spark-requirement.txt file:

azureml-dataprep[blobfuse]==2.4.0
fusepy==3.0.1
pyspark==2.4.4
pandas==0.25.3
pip==19.3.1
numpy==1.18.1

packages in my local environment

And here is a snippet of my pipeline with the spark config and PythonScriptStep:

# Environment set up for PySpark Compute
spark_env = plf.get_environment(env_name='spark_env',
                                req_path=os.path.join(os.getcwd(), 'compute/aml_config/spark-requirements.txt'),
                                enable_docker=True,
                                docker_base_image='microsoft/mmlspark:0.16',
                                url=index_url + ' ' + mlpackage_req)

# use pyspark framework
spark_run_config = RunConfiguration(framework="pyspark")
spark_run_config.environment = spark_env

roll_step = PythonScriptStep(
    name='roll.py',
    script_name='roll.py',
    arguments=['--input_dir', joined_data,
                '--output_dir', rolled_data,
                '--script_dir', ".",
                '--min_date', '2015-06-30',
                '--pct_rank', 'True'],
    compute_target=compute_target_spark,
    inputs=[joined_data],
    outputs=[rolled_data],
    runconfig=spark_run_config,
    source_directory=os.path.join(os.getcwd(), 'compute', 'roll'),
    allow_reuse=pipeline_reuse
)

從 spark-requirements.txt 環境和 microsoft/mmlspark:0.16 基礎映像開始，然後透過 azureml/data/context_managers.py 和 azureml/data/_dataprep_helper.py 追蹤故障。使用 roll.py 重現 libfuse 匯入錯誤，並判斷該相依性應放在映像中還是 Python 環境中。明確確認所需的相依性和經過驗證的設定即表示完成。