galaxyproject / galaxyproject/pulsar
JSONDecodeError on `launch_config` requires manual intervention
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 46
- Forks
- 62
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 15
Description
Occasionally my jobs volume will fill up such that launch_config files can be created but are empty. This results in the following looping error for any such jobs:
2025-09-24 14:54:03,105 ERROR [pulsar.managers.stateful][[manager=vgp_jetstream2]-[action=monitor]] Failed checking active job status for job_id 70361127
Traceback (most recent call last):
File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/pulsar/managers/stateful.py", line 382, in _monitor_active_jobs
self._check_active_job_status(active_job_id)
File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/pulsar/managers/stateful.py", line 396, in _check_active_job_status
self.stateful_manager.get_status(active_job_id)
File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/pulsar/managers/stateful.py", line 165, in get_status
proxy_status, state_change = self.__proxy_status(job_directory, job_id)
File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/pulsar/managers/stateful.py", line 187, in __proxy_status
proxy_status = job_directory.load_metadata(JOB_FILE_FINAL_STATUS)
File "/srv/pulsar/main/venv/lib64/python3.9/site-packages/pulsar/managers/base/__init__.py", line 367, in load_metadata
return json.loads(contents.decode())
File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
To fix, I remove the active-jobs files for affected jobs and requeue them. A more graceful way to recover from this would be ideal.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pulsar/managers/stateful.py at _monitor_active_jobs, _check_active_job_status, and __proxy_status, then inspect load_metadata in pulsar/managers/base/init.py. Reproduce the empty metadata-file case and determine how recovery should avoid the repeated JSONDecodeError. Done means affected jobs recover without manually removing active-jobs files and requeueing them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100