Very large Ceph RadosGW S3 object staging bug

Bug report

Expected behavior and actual behavior

I am running a pipeline using the k8s executor from inside a k8s pod (I know this isn't generally recommended but it actually works very well) and use an S3 URI (S3 API mocked via Ceph RadosGW) as a file input. Before any k8s pods are deployed the S3 URI is staged to the work directory at which point for very large objects (> ~120G from my testing) an extremely generic java aws exception is raised killing the pipeline: com.amazonaws.SdkClientException: Unable to store object contents to disk: Connection reset by peer.

Smaller files stage absolutely fine which is the expected behaviour.

I'm not 100% certain this is actually a problem with nextflow or the nf-amazon plugin but I was hoping you might be able to shed some light

Steps to reproduce the problem

No complex reproduction script required, simply try to stage an extremely large file (if you cannot reproduce that would be valuable too since it would indicate the issue is something to do with our setup)

Program output

Lightly redacted nextflow.log available below
nextflow-Copy1 (1).log

Environment

• Nextflow version: 23.10.0
• Java version: openjdk 21.0.1 2023-10-17
• Operating system: Linux
• Bash version: GNU bash, version 5.2.21(1)-release (x86_64-alpine-linux-musl)

Additional context

The pipe is also being executed as a python subprocess, I'm aware this sounds horrendous but again, it actually works extremely reliably.

Start by reproducing S3 URI staging through the k8s executor with a very large object in Ceph RadosGW, then inspect the attached nextflow log and the AWS exception. Done means the object stages successfully, or the failure is narrowed to Nextflow, the nf-amazon plugin, AWS SDK, or the Ceph setup.