Azure nodes keep becoming unusable due to lack of space when using Docker images to run jobs on Azure

Bug report

Expected behavior and actual behavior

Expected: Azure nodes remain usable without manual intervention
Actual: Azure nodes keep becoming unusable due to lack of space

Steps to reproduce the problem

Not easily possible. Would require a multi-process config that specified multiple, application-specific docker images (see description below)

Program output

There is no output as the nodes become unusable due to lack of space, and nextflow doesn't detect that they are in this state

Environment

• Nextflow version: 25.04.7
• Java version: java 17.0.1 2021-10-19 LTS
• Operating system: linux
• Bash version: GNU bash, version 5.2.21(1)-release (x86_64-pc-linux-gnu)

Additional context

Note, this is a duplicate of an issue created on the nf-azure plugin page, but I'm not sure who is monitoring that.

Azure nodes get into an "unusable" state b/c their disks are considered to be full. Forensics on the nodes indicates that the "overlay" drives on the nodes are getting filled, and become unwriteable, causing the jobs to hang. Note, the Azure pool isn't run continuously. It is instantiated and deleted by the pipeline upon pipeline trigger and completion.

The only solution is to manually reboot the nodes via the webUI. Our working theory is that because our pipeline uses job/process-specific docker images, the /var/lib/docker drives are getting filled by the different docker images.

For that reason, we'd like to be able to trigger a docker prune command to remove the docker image upon process completion, but that doesn't seem possible at this point.

These are the fix attempts that we've tried so far (that haven't resolved the issue):
** updated all of our processes to use a temp directory that's local to the working directory

**added the --rm parameter to the docker.runOptions, as well as the following --storage-opt size=100G AND also specified docker.remove = true

**added a command to prune the docker images in the afterScript directive, i.e.
process.afterScript = "docker system prune -a -f"

However, this only causes all the jobs to fail because they are unable to find the docker command.

Upon investigation, the actual docker command that is used to launch the docker image isn't available in the .command.run file, so it unclear on how to trigger docker in order to prune the images upon job completion.

Any suggestions on how to have Azure nodes remove docker images upon process completion?

The report names Azure nodes, process-specific Docker images, afterScript, and .command.run, but no source files or tests. Start by tracing how Docker images are launched and cleaned up for a multi-process configuration, then verify that completed jobs do not fill node storage and that subsequent jobs remain usable.