Azure Batch: custom `startTask.script` never executes when `copyToolInstallMode = 'node'`

Bug report

Expected behavior and actual behavior

Expected: When configuring azure.batch.pools.<name>.startTask.script, the custom script should execute on the compute node during pool start task, alongside the azcopy installation (when copyToolInstallMode = 'node', the default).

Actual: The custom startTask.script never executes. Only the azcopy installation command runs. This occurs regardless of whether the azcopy installation succeeds or fails, suggesting a command construction/quoting issue rather than && short-circuit logic.

Steps to reproduce the problem

nextflow.config:

process {
    executor = 'azurebatch'
    machineType = 'Standard_E2ds_v5'
    queue = 'auto'
}

azure {
    storage {
        accountName = 'myaccount'
        accountKey = env("AZURE_STORAGE_ACCOUNT_KEY")
    }
    batch {
        location = 'eastus'
        accountKey = env("AZURE_BATCH_ACCOUNT_KEY")
        accountName = 'mybatch'
        autoPoolMode = true
        allowPoolCreation = true
        pools {
            'auto' {
                autoScale = false
                vmCount = 1
                sku = "batch.node.ubuntu 24.04"
                offer = "ubuntu-hpc"
                publisher = "microsoft-dsvm"
                startTask {
                    privileged = true
                    script = 'echo "CUSTOM SCRIPT EXECUTED"'
                }
            }
        }
    }
}

Run any pipeline (e.g. nextflow run hello). The echo never appears in the start task output.

Program output

The generated start task command line on the node is:

bash -c "chmod +x azcopy && mkdir $AZ_BATCH_NODE_SHARED_DIR/bin/ && cp azcopy $AZ_BATCH_NODE_SHARED_DIR/bin/" && bash -c 'echo "CUSTOM SCRIPT EXECUTED"'

The second bash -c '...' block never executes. Adding debug output (e.g. echo, writing to a marker file) at the start of the custom script confirms it is never reached.

Environment

• Nextflow version: 25.04.3
• Java version: 21
• Operating system: Azure Batch (Linux nodes, microsoft-dsvm/ubuntu-hpc 24.04)

Additional context

The issue is in AzBatchService.createStartTask(). Two bash -c commands with different quoting styles (double quotes for azcopy, single quotes for user script) are joined with &&:

// azcopy install (double-quoted)
startCmd << 'bash -c "chmod +x azcopy && mkdir \$AZ_BATCH_NODE_SHARED_DIR/bin/ && cp azcopy \$AZ_BATCH_NODE_SHARED_DIR/bin/"'

// user script (single-quoted)
startCmd << "bash -c '${opts.script.replace(/'/,/''/)}'".toString()

// joined
return new BatchStartTask(startCmd.join(' && '))

Since Azure Batch invokes the start task command line through a shell, the mixed quoting between the two bash -c invocations likely causes the second command to be silently dropped or misinterpreted.

Workaround: Set azure.batch.copyToolInstallMode = 'off' and install azcopy manually within the startTask.script:

azure.batch.copyToolInstallMode = 'off'

startTask {
    privileged = true
    script = '''
        # Your custom setup here
        echo "CUSTOM SCRIPT EXECUTED"

        # Manual azcopy install
        mkdir -p $AZ_BATCH_NODE_SHARED_DIR/bin
        wget -q https://github.com/nextflow-io/azcopy-tool/raw/linux_amd64_10.8.0/azcopy -O $AZ_BATCH_NODE_SHARED_DIR/bin/azcopy
        chmod +x $AZ_BATCH_NODE_SHARED_DIR/bin/azcopy
    '''.stripIndent()
}

Suggested fix: Wrap both commands in a single bash -c invocation, or use ; instead of && so the commands are independent, or reverse the order so user scripts run first.

Start in plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy at createStartTask(), then compare the generated command with the reported output. Reproduce with the provided nextflow.config and nextflow run hello using node copy-tool installation. Done means both the azcopy installation and the configured custom startTask.script execute on the compute node.