galaxyproject / galaxyproject/pulsar

Pulsar queued_python potentially isn't executing jobs on my machine.

Open
#375 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
46
Forks
62
Avg merge
3d 15h
Merged PRs (30d)
14

Description

Reported in admins matrix,

app.yml

```
---
dependency_resolution:
resolvers:
- auto_init: true
auto_install: true
type: conda
job_metrics_config_file: job_metrics_conf.yml
min_polling_interval: 0.5
persistence_directory: /mnt/pulsar/files/persisted_data
private_token: asdf
staging_directory: /mnt/pulsar/files/staging
tool_dependency_dir: /mnt/pulsar/deps
managers:
_default_:
type: queued_python
num_concurrent_jobs: 1
```

but my jobs don't execute:

```
Oct 09 16:51:03 worker1 uwsgi[66861]: 2024-10-09 16:51:03,445 DEBUG [pulsar.managers.base][uWSGIWorker1Core0] job_id: 16 - checking tool file cutWrapper.pl
Oct 09 16:51:03 worker1 uwsgi[66861]: 2024-10-09 16:51:03,446 DEBUG [galaxy.tool_util.deps][uWSGIWorker1Core0] Using dependency perl version 5.26 of type conda
Oct 09 16:51:03 worker1 uwsgi[66861]: [pid: 66861|app: 0|req: 610/610] 145.38.195.22 () {32 vars in 5437 bytes} [Wed Oct 9 16:51:03 2024] POST /managers/_default_/jobs/16/submit?command_line=%2Fbin%2Fbash+%2Fm..........
```

Nate suggested py-spy

```
root@worker1:/mnt/pulsar# py-spy dump --pid 66861
Process 66861: /mnt/pulsar/venv/bin/uwsgi --ini-paste /mnt/pulsar/config/server.ini
Python v3.10.12 (/mnt/pulsar/venv/bin/uwsgi)

Thread 0x7F93FB7D1040 (active): "uWSGIWorker1Core0"
Thread 0x7F93F35FF640 (idle): "Thread-1 (run_next)"
wait (threading.py:320)
get (queue.py:171)
run_next (pulsar/managers/queued.py:83)
run (threading.py:953)
_bootstrap_inner (threading.py:1016)
_bootstrap (threading.py:973)
```

and I can verify that things get added to the queue, but nothing seems to be read from the queue.

```
pulsar-check --private_token=asdf --debug


INFO:pulsar.client.manager:Setting Pulsar client class to standard, non-caching variant.

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/t/script.py] (action_type: [transfer])

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/dataset_0.dat] (action_type: [transfer])

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/dataset_0_files/input_subdir/extra] (action_type: [transfer])

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/metadata/12312231231231.dat] (action_type: [transfer])

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/w/config.txt] (action_type: [transfer])

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/m/metadata_test123] (action_type: [transfer])

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/idx/seq/human_full_seqs] (action_type: [transfer])

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/idx/bwa/human.fa.fai] (action_type: [transfer])

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/idx/bwa/human.fa] (action_type: [transfer])

DEBUG:pulsar.client.client:Uploading path [/tmp/pulsar-check-client.dn8jifhl/w/config.txt] (action_type: [message])

```

swapping to `queued_condor` and making no other changes, enabled jobs to execute.

Running the latest pulsar:

```
(venv) root@worker1:/home/hrasche2# pip freeze | grep pulsar
pulsar-app==0.15.6
```

In this case I'd rather not install htcondor if it isn't necessary.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.