galaxyproject / galaxyproject/pulsar
Recommended settings for shared filesystem setup
- Dominant language
- Python
- Stars
- 46
- Forks
- 62
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 15
Description
I have some difficulties setting up, what I think is a basic pulsar setup. I want to run a specific local tool with e.g. `python $__tool_directory__/tool.py` on a remote system. The remote system has the same shared filesystem as the galaxy server and I can run pulsar with the same user. I tried with `default_file_action=none` but then the `tool.py` is not found. With `remote_copy` that part works, but the input files gets unnecessarily staged.. In addition, with this setting I get the following error:
```
Traceback (most recent call last):
File "/tmpdata/pulsar/files/staging/145701/tool_files/dataset_export.py", line 32, in
with open(script_path / "config.yaml") as c:
FileNotFoundError: [Errno 2] No such file or directory: '/tmpdata/pulsar/files/staging/145701/tool_files/config.yaml
```
This is set in the pulsar app.yml:
```
conda_auto_init: true
conda_auto_install: true
staging_directory: /tmpdata/pulsar/files/staging
persistence_directory: /tmpdata/pulsar/files/persisted_data
tool_dependency_dir: /tmpdata/pulsar/dependencies
```
This in galaxy's `job_conf.xml`
```
http://remote:8913/
xxx
remote_copy
```
What would be the right settings for my setup?
Contributor guide
Research direction
Start with the Pulsar app.yml and Galaxy job_conf.xml settings shown in the report, focusing on default_file_action, staging_directory, and the remote_copy destination. Reproduce or trace the staging behavior around the reported FileNotFoundError; done means identifying settings that make tool.py and config.yaml available without unnecessarily staging shared input files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100