K8s executor generates .command.run with hardcoded host paths on macOS with Minikube

Bug report

I'm trying to run a very simple hello world example workflow on a local minikube cluster. My nf script is the following:

nextflow.enable.dsl=2

process sayHello {
    input:
    val name

    script:
    """
    echo "Hello $name" > hello_nf.txt
    """
}


workflow {
    main:
        sayHello("Kubernetes World")
}

My config file is:

process {
    executor = 'k8s'
    container = 'ubuntu' 
}

k8s {
    namespace = 'default'
    storageClaimName = 'my-pvc'
    storageMountPath = '/workspace'
    autoMountHostPaths = false 
}

I have a pvc.yaml file to define the PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

which I applied using: kubectl apply -f pvc.yaml. I am using minikube with the docker container.

Expected behavior and actual behavior

I expected a pod to be created to write the hello message to a file hello_nf.txt. Instead the pod logs show that a file on the local host was attempted to be accessed:

/bin/bash: <local_path>/work/2b/ca74d55072b8df760bea394d415def/.command.run: No such file or directory

I can see in the generated .command.run file that it is always using a local path instead of the path defined by k8s.storageMountPath.

Steps to reproduce the problem

• Apply the PVC: kubectl apply -f pvc.yaml
• Run the nf workflow: nextflow run main.nf -c nextflow.config

Program output

Here's the output from kubectl pod describe:

Name: nf-f8afd217cfd01aa5dc500522b4c55426-7b6a9
Namespace: default
Priority: 0
Service Account: default
Node: minikube/192.168.49.2
Start Time: Mon, 14 Apr 2025 16:50:13 +0100
Labels: nextflow.io/app=nextflow
nextflow.io/processName=sayHello
nextflow.io/runName=prickly_dijkstra
nextflow.io/sessionId=uuid-11a5914a-4b3c-4c8f-8749-04711f9798d0
nextflow.io/taskName=sayHello
Annotations: <none>
Status: Failed
IP: 10.244.0.5
IPs:
IP: 10.244.0.5
Containers:
nf-f8afd217cfd01aa5dc500522b4c55426-7b6a9:
Container ID: docker://9004b49455037f61019ff305ad82090dd5b33136df9b2b67051940cd4532bf57
Image: ubuntu
Image ID: docker-pullable://ubuntu@sha256:1e622c5f073b4f6bfad6632f2616c7f59ef256e96fe78bf6a595d1dc4376ac02
Port: <none>
Host Port: <none>
Args:
/bin/bash
-ue
<local_path>/work/f8/afd217cfd01aa5dc500522b4c55426/.command.run
State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 14 Apr 2025 16:50:20 +0100
Finished: Mon, 14 Apr 2025 16:50:20 +0100
Ready: False
Restart Count: 0
Requests:
cpu: 1
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hvcxv (ro)
/workspace from vol-1 (rw)
Conditions:
Type Status
PodReadyToStartContainers False
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
vol-1:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: my-pvc
ReadOnly: false
kube-api-access-hvcxv:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 57s default-scheduler Successfully assigned default/nf-f8afd217cfd01aa5dc500522b4c55426-7b6a9 to minikube
Normal Pulling 57s kubelet Pulling image "ubuntu"
Normal Pulled 50s kubelet Successfully pulled image "ubuntu" in 6.22s (6.22s including waiting). Image size: 100640822 bytes.
Normal Created 50s kubelet Created container: nf-f8afd217cfd01aa5dc500522b4c55426-7b6a9
Normal Started 50s kubelet Started container nf-f8afd217cfd01aa5dc500522b4c55426-7b6a9

And the generated .command.run file looks like:

#!/bin/bash
NXF_CHDIR=<local_path>/work/2b/ca74d55072b8df760bea394d415def
### ---
### name: 'sayHello'
### container: 'ubuntu'
### ...
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x
NXF_ENTRY=${1:-nxf_main}


nxf_sleep() {
  sleep $1 2>/dev/null || sleep 1;
}

nxf_date() {
    local ts=$(date +%s%3N);
    if [[ ${#ts} == 10 ]]; then echo ${ts}000
    elif [[ $ts == *%3N ]]; then echo ${ts/\%3N/000}
    elif [[ $ts == *3N ]]; then echo ${ts/3N/000}
    elif [[ ${#ts} == 13 ]]; then echo $ts
    else echo "Unexpected timestamp value: $ts"; exit 1
    fi
}

nxf_env() {
    echo '============= task environment ============='
    env | sort | sed "s/\(.*\)AWS\(.*\)=\(.\{6\}\).*/\1AWS\2=\3xxxxxxxxxxxxx/"
    echo '============= task output =================='
}

nxf_kill() {
    declare -a children
    while read P PP;do
        children[$PP]+=" $P"
    done < <(ps -e -o pid= -o ppid=)

    kill_all() {
        [[ $1 != $$ ]] && kill $1 2>/dev/null || true
        for i in ${children[$1]:=}; do kill_all $i; done
    }

    kill_all $1
}

nxf_mktemp() {
    local base=${1:-/tmp}
    mkdir -p "$base"
    if [[ $(uname) = Darwin ]]; then mktemp -d $base/nxf.XXXXXXXXXX
    else TMPDIR="$base" mktemp -d -t nxf.XXXXXXXXXX
    fi
}

nxf_fs_copy() {
  local source=$1
  local target=$2
  local basedir=$(dirname $1)
  mkdir -p $target/$basedir
  cp -fRL $source $target/$basedir
}

nxf_fs_move() {
  local source=$1
  local target=$2
  local basedir=$(dirname $1)
  mkdir -p $target/$basedir
  mv -f $source $target/$basedir
}

nxf_fs_rsync() {
  rsync -rRl $1 $2
}

nxf_fs_rclone() {
  rclone copyto $1 $2/$1
}

nxf_fs_fcp() {
  fcp $1 $2/$1
}

on_exit() {
    exit_status=${nxf_main_ret:=$?}
    printf -- $exit_status > <local_path>/work/2b/ca74d55072b8df760bea394d415def/.exitcode
    set +u
    exit $exit_status
}

on_term() {
    set +e
    [[ "$pid" ]] && nxf_kill $pid
}

nxf_launch() {
    /bin/bash -ue <local_path>/work/2b/ca74d55072b8df760bea394d415def/.command.sh
}

nxf_stage() {
    true
}

nxf_unstage() {
    true
    [[ ${nxf_main_ret:=0} != 0 ]] && return
}

nxf_main() {
    trap on_exit EXIT
    trap on_term TERM INT USR2
    trap '' USR1

    [[ "${NXF_CHDIR:-}" ]] && cd "$NXF_CHDIR"
    NXF_SCRATCH=''
    [[ $NXF_DEBUG > 0 ]] && nxf_env
    touch <local_path>/work/2b/ca74d55072b8df760bea394d415def/.command.begin
    set +u
    set -u
    [[ $NXF_SCRATCH ]] && cd $NXF_SCRATCH
    export NXF_TASK_WORKDIR="$PWD"
    nxf_stage

    set +e
    (set -o pipefail; (nxf_launch | tee .command.out) 3>&1 1>&2 2>&3 | tee .command.err) &
    pid=$!
    wait $pid || nxf_main_ret=$?
    nxf_unstage
}

$NXF_ENTRY

Note that I have replaced the path to my local working directory with <local_path> throughout these. I am also attaching the .nextflow.log file.

.nextflow.log

Environment

• Nextflow version: 24.10.5.5935
• Java version:
  openjdk 17.0.10 2024-01-16
  OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
  OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode)
• Operating system: macOS 15.3.2
• Bash version: zsh 5.9 (arm64-apple-darwin24.0)

Reproduce with main.nf, nextflow.config, pvc.yaml, and nextflow run main.nf -c nextflow.config, then inspect the generated .command.run and the attached .nextflow.log. Compare the host paths in the script with the pod's /workspace mount; done means the Minikube pod runs the workflow successfully without trying to access a local host path.