galaxyproject / galaxyproject/pulsar

Implement periodic checking in on specific jobs.

Open
#66 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
46
Forks
62
Avg merge
3d 9h
Merged PRs (30d)
15

Description

- Managers should monitor the MQ for a new topic (`send_update` or something).
- Message should include a list of job ids to report on.
- For each job id, an `update` should be fired reporting the status or `lost` if Pulsar is no longer aware of the job.
- Galaxy should be configured to periodically send this request.

Galaxy with its "synchronicity", "actual database", "years or maturity", etc... is pretty good at not loosing track of jobs - so @natefoo and I decided it would be good if there was some way to periodically have Galaxy poll the jobs it believes to be open so if messages are lost or mishandled in transit the jobs do not stay open indefinitely on the Galaxy side.

Contributor guide

Open the contributing guide

Research direction

Start by locating the manager MQ consumers and the existing handling for job status updates, then find how Galaxy configures polling of open jobs. Trace the current update and lost-job paths and determine where a periodic request containing job IDs can fit. Done means the request is handled for each listed job and Galaxy can periodically trigger it without leaving lost jobs open.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.