galaxyproject / galaxyproject/pulsar

Recommended settings for shared filesystem setup

Open
#209 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
46
Forks
62
Avg merge
3d 9h
Merged PRs (30d)
15

Description

I have some difficulties setting up, what I think is a basic pulsar setup. I want to run a specific local tool with e.g. `python $__tool_directory__/tool.py` on a remote system. The remote system has the same shared filesystem as the galaxy server and I can run pulsar with the same user. I tried with `default_file_action=none` but then the `tool.py` is not found. With `remote_copy` that part works, but the input files gets unnecessarily staged.. In addition, with this setting I get the following error:
```
Traceback (most recent call last):
File "/tmpdata/pulsar/files/staging/145701/tool_files/dataset_export.py", line 32, in
with open(script_path / "config.yaml") as c:
FileNotFoundError: [Errno 2] No such file or directory: '/tmpdata/pulsar/files/staging/145701/tool_files/config.yaml
```

This is set in the pulsar app.yml:
```
conda_auto_init: true
conda_auto_install: true
staging_directory: /tmpdata/pulsar/files/staging
persistence_directory: /tmpdata/pulsar/files/persisted_data
tool_dependency_dir: /tmpdata/pulsar/dependencies
```

This in galaxy's `job_conf.xml`
```

http://remote:8913/
xxx
remote_copy

```

What would be the right settings for my setup?

Contributor guide

Open the contributing guide

Research direction

Start with the Pulsar app.yml and Galaxy job_conf.xml settings shown in the report, focusing on default_file_action, staging_directory, and the remote_copy destination. Reproduce or trace the staging behavior around the reported FileNotFoundError; done means identifying settings that make tool.py and config.yaml available without unnecessarily staging shared input files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.