nextflow-io / nextflow-io/nextflow

Google Batch: share a single getTaskStatus lookup in getTraceRecord()

Open
#7,636 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Background

Follow-up recorded during the review of #7629 (non-blocking there, tracked here).

GoogleBatchTaskHandler.getTraceRecord() makes two separate client.getTaskStatus(jobId, taskId) calls per task, both fetching the same object:

  • getMachineInfo()updateZoneFromEvents()client.getTaskStatus(...) to resolve the actual execution zone from the status events
  • getNumSpotInterruptions(jobId)client.getTaskStatus(...) to count preemption events (exit code 50001)

Both read statusEventsList off the same response.

Why it matters now

Before #7629, array-child handlers had a null machineInfo, so getMachineInfo() returned early and never reached updateZoneFromEvents() — only the spot-interruption lookup ran. Now that children carry a non-null machineInfo, they perform the zone resolution too. That is the desired behaviour (each child resolves its own actual zone rather than inheriting the parent's planned one), but it doubles the number of Batch API calls at trace time. On a large array job that is one extra getTaskStatus() per task.

Proposal

Fetch the task status once in getTraceRecord() and share it between the zone resolution and the spot-interruption count, e.g. pass the already-fetched status (or its statusEventsList) into both paths instead of having each one call the client. resolveZoneFromEvents(List<StatusEvent>) is already factored to take events directly, so the zone side is a small change; getNumSpotInterruptions() would need an overload accepting the status/events.

Both call sites already tolerate a null status, so the shared lookup should keep the same defensive handling.

References

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy by reading getTraceRecord(), updateZoneFromEvents(), and getNumSpotInterruptions(). Trace how statusEventsList is consumed and check the existing Google Batch handler tests, if available. Done means each task performs one status lookup while preserving zone resolution, interruption counting, and null handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, groovy
Domain
cloud
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.