nextflow-io / nextflow-io/nextflow

Google Batch: Fusion scratch mount omitted for machines with built-in Local SSDs (-lssd)

Open
#7,627 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

The Google Batch executor omits the Fusion scratch mount when selecting a machine with built-in Local SSDs, such as c3d-standard-8-lssd.

Expected behavior and actual behavior

Expected:

  • Fusion should be able to use the machine's built-in Local SSD storage as scratch storage, with the appropriate /tmp volume mount configured.
    Actual:
  • The allocation builder removes the explicit scratch-disk declaration and returns requiresScratchVolume=false. The generated job request does not include the scratch volume mounted at /tmp.
Steps to reproduce the problem

This was reproduced offline against the actual plugin allocation builder.

Configure Google Batch with Fusion and a Local SSD-equipped machine:

process {
    executor = 'google-batch'
    machineType = 'c3d-standard-8-lssd'
    disk = [request: 375.GB, type: 'local-ssd']
    scratch = false
}

fusion.enabled = true
wave.enabled = true
  • Invoke GoogleBatchTaskHandler.buildInstancePolicyOrTemplate() with this configuration and machine selection stubbed to return the specified machine.
  • Inspect the resulting disk declarations and requiresScratchVolume flag.
  • The same behavior occurs when disk is omitted and Fusion automatically requests Local SSD scratch. Using n2-standard-8 provides a control case that retains the disk declaration and scratch mount.
Program output
    n2-standard-8: attachedDisks=1, requiresScratchVolume=true
    c3-standard-8-lssd: attachedDisks=0, requiresScratchVolume=false
    c3d-standard-8-lssd: attachedDisks=0, requiresScratchVolume=false
    c4-standard-16-lssd: attachedDisks=0, requiresScratchVolume=false
    c4d-standard-16-lssd: attachedDisks=0, requiresScratchVolume=false
Environment
  • Nextflow version: 26.04.6
  • Google plugin version: nf-google@1.27.3
  • Java version: OpenJDK 17.0.13 (Temurin)
  • Operating system: macOS
  • Bash version: 5.3
Additional context
  • GoogleBatchMachineTypeSelector.findValidLocalSSDSize() returns zero for machines classified by notConfigurableLocalSSD().
  • GoogleBatchTaskHandler.buildInstancePolicyOrTemplate() then replaces the disk with an untyped zero-size request. This skips the block that declares the scratch disk and sets requiresScratchVolume=true.
  • The same code path is present in master at c63383c11.
  • My workaround is to use non--lssd machine variants with dedicated hyperdisk-balanced scratch storage but would prefer to keep using SSDs on all Batch workers.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at GoogleBatchMachineTypeSelector.findValidLocalSSDSize() and GoogleBatchTaskHandler.buildInstancePolicyOrTemplate(), comparing the -lssd path with the n2-standard-8 control case. Verify the builder output retains the Local SSD declaration, sets requiresScratchVolume=true, and includes the /tmp scratch mount for built-in Local SSD machines.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, groovy
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.