nextflow-io / nextflow-io/nextflow
Google Batch: Fusion scratch mount omitted for machines with built-in Local SSDs (-lssd)
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
The Google Batch executor omits the Fusion scratch mount when selecting a machine with built-in Local SSDs, such as c3d-standard-8-lssd.
Expected behavior and actual behavior
Expected:
- Fusion should be able to use the machine's built-in Local SSD storage as scratch storage, with the appropriate /tmp volume mount configured.
Actual: - The allocation builder removes the explicit scratch-disk declaration and returns
requiresScratchVolume=false. The generated job request does not include the scratch volume mounted at/tmp.
Steps to reproduce the problem
This was reproduced offline against the actual plugin allocation builder.
Configure Google Batch with Fusion and a Local SSD-equipped machine:
process {
executor = 'google-batch'
machineType = 'c3d-standard-8-lssd'
disk = [request: 375.GB, type: 'local-ssd']
scratch = false
}
fusion.enabled = true
wave.enabled = true
- Invoke
GoogleBatchTaskHandler.buildInstancePolicyOrTemplate()with this configuration and machine selection stubbed to return the specified machine. - Inspect the resulting disk declarations and
requiresScratchVolumeflag. - The same behavior occurs when
diskis omitted and Fusion automatically requests Local SSD scratch. Usingn2-standard-8provides a control case that retains the disk declaration and scratch mount.
Program output
n2-standard-8: attachedDisks=1, requiresScratchVolume=true
c3-standard-8-lssd: attachedDisks=0, requiresScratchVolume=false
c3d-standard-8-lssd: attachedDisks=0, requiresScratchVolume=false
c4-standard-16-lssd: attachedDisks=0, requiresScratchVolume=false
c4d-standard-16-lssd: attachedDisks=0, requiresScratchVolume=false
Environment
- Nextflow version: 26.04.6
- Google plugin version: nf-google@1.27.3
- Java version: OpenJDK 17.0.13 (Temurin)
- Operating system: macOS
- Bash version: 5.3
Additional context
GoogleBatchMachineTypeSelector.findValidLocalSSDSize()returns zero for machines classified bynotConfigurableLocalSSD().GoogleBatchTaskHandler.buildInstancePolicyOrTemplate()then replaces the disk with an untyped zero-size request. This skips the block that declares the scratch disk and setsrequiresScratchVolume=true.- The same code path is present in master at
c63383c11. - My workaround is to use non-
-lssdmachine variants with dedicatedhyperdisk-balancedscratch storage but would prefer to keep using SSDs on all Batch workers.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at GoogleBatchMachineTypeSelector.findValidLocalSSDSize() and GoogleBatchTaskHandler.buildInstancePolicyOrTemplate(), comparing the -lssd path with the n2-standard-8 control case. Verify the builder output retains the Local SSD declaration, sets requiresScratchVolume=true, and includes the /tmp scratch mount for built-in Local SSD machines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, groovy
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100