galaxyproject / galaxyproject/pulsar

Batch co-execution issue tracker

Open
#465 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
46
Forks
62
Avg merge
3d 9h
Merged PRs (30d)
15

Description

This issue tracks the changes I made to benchmark the co-execution runner. I have grouped related changes to reduce the number of pull requests while also keeping each relatively easy to review and understand.

I will fill in the actual PR numbers once I open them.

PR 1 — Co-execution deadlocks & output collection (critical)

The Pulsar co-execution runner uses two runnables: one that runs the tool itself, and a second sidecar that stages input data and collects results. Both were run as foreground runnables (the default) and GCP Batch runs foreground runnable in sequence waiting for the first to complete before starting the next.

This PR fixes the sidecar/tool runnable ordering so both containers run, collects outputs written to the working/ directory (including UUID-named datasets), re-raises genuine infrastructure errors instead of swallowing them, and passes docker_extra_volumes (CVMFS) through to the Batch containers.

PR1.1 — Re-raise infrastructure errors.

Split out from #466 and now just re-raises infrastructure errors (ImportError, MemoryError, SystemError, and infrastructure OSErrors; disk full, I/O errors) so they fail the job . FileNotFound and 403 errors still flow through.

PR 2 — Custom VM boot disk image & size

Adds custom_vm_image and boot_disk_size_gb to GcpJobParams so jobs can boot
from a custom image (e.g. one with CVMFS pre-installed) and size the boot disk to fit it. Pairs with a small companion change in Galaxy's PulsarGcpBatchJobRunner (PASSTHROUGH_PARAMS) so the image URI can be set once at the runner level.

PR 3 — Dynamic VM sizing & job lifecycle

Sizes the VM from TPV cores/mem, injects GALAXY_SLOTS/GALAXY_MEMORY_MB and sets the container compute resource so tools use the whole VM, validates local-SSD counts for N2/N2D machine types, generates collision-free job names with a configurable job_id_prefix, and makes job deletion on cancel optional (plus a kill() for the AMQP client) so jobs can be retained for debugging.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

This is a tracking issue rather than a standalone first contribution; start by reviewing PRs #466, #467, #472, and #473. The relevant entry points are the co-execution runner, GcpJobParams, PulsarGcpBatchJobRunner, and the AMQP client; the work is done when the grouped changes are resolved through those PRs.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.