msg="archive/tar: write too long

### Pre-requisites

- [x] I have double-checked my configuration
- [X] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [X] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))

### What happened? What did you expect to happen?

i use artifacts to transfer a directory from step generate-artifact to consume-artifact, while the folder is quite large, around 800MB, the wait container reports an error as follows. It seems that this method does not support transferring large files, or my configuration is incorrect. How should I resolve this issue?

### Version(s)

v3.5.8

### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

```YAML
apiVersion: v1
kind: PersistentVolume
metadata:
labels:
tmp: tmp-pv
name: pipeline-684-model-684-pv
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 50Pi
csi:
driver: csi.juicefs.com
fsType: juicefs
nodePublishSecretRef:
name: tdsc-sc-secret
namespace: tdsc-csi
volumeAttributes:
csi.storage.k8s.io/provisioner-secret-name: tdsc-sc-secret
juicefs/clean-cache: 'false'
csi.storage.k8s.io/node-publish-secret-name: tdsc-sc-secret
juicefs/mount-cpu-limit: '5'
juicefs/mount-mem-limit: 5Gi
csi.storage.k8s.io/provisioner-secret-namespace: tdsc-csi
juicefs/mount-image: xxx
juicefs/mount-cpu-request: '1'
juicefs/mount-mem-request: '5'
csi.storage.k8s.io/node-publish-secret-namespace: tdsc-csi
volumeHandle: pipeline-684-model-684-pv
mountOptions:
- subdir=shared/1/1/1/jobtaskmgr/traintuning/683/
- no-bgjob
- no-session
- cache-size=0
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pipeline-684-model-684-pvc
namespace: ns-1
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Pi
storageClassName: ''
volumeMode: Filesystem
volumeName: pipeline-684-model-684-pv
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: artifact-passing-1
namespace: ns-1
spec:
automountServiceAccountToken: true
entrypoint: artifact-example
templates:
- name: artifact-example
steps:
- - name: generate-artifact
template: hello-world-to-file
- - name: consume-artifact
template: print-message-from-file
arguments:
artifacts:
- name: message
from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"
- name: hello-world-to-file
archiveLocation:
archiveLogs: false
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nvidia.com/gpu.product
operator: In
values:
- NVIDIA-A10
- key: nvidia-device-enable
operator: DoesNotExist
container:
image: xxx
command: [/bin/bash, -c]
args: ["echo hello world | tee /tmp/hello_world.txt"]
volumeMounts:
- mountPath: /workspace/model-detect
name: pipeline-684-model-684-volume
outputs:
artifacts:
- name: hello-art
path: /workspace/model-detect
volumes:
- name: pipeline-684-model-684-volume
persistentVolumeClaim:
claimName: pipeline-684-model-684-pvc
- name: print-message-from-file
archiveLocation:
archiveLogs: false
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nvidia.com/gpu.product
operator: In
values:
- NVIDIA-A10
- key: nvidia-device-enable
operator: DoesNotExist
inputs:
artifacts:
- name: message
path: /workspace/model-detect
- name: objects
path: /tmp/
s3:
key: private/1/1/1/jobtaskmgr/package/689
container:
image: xxx
command: [/bin/bash, -c]
args: ["sleep 10000000s"]
```

### Logs from the workflow controller

```text
time="2025-01-09T05:56:11.823Z" level=info msg="Processing workflow" Phase= ResourceVersion=46349677 namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.829Z" level=info msg="resolved artifact repository" artifactRepositoryRef="ns-1/#"
time="2025-01-09T05:56:11.829Z" level=info msg="Task-result reconciliation" namespace=ns-1 numObjs=0 workflow=artifact-passing-1
time="2025-01-09T05:56:11.829Z" level=info msg="Updated phase -> Running" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.829Z" level=info msg="Creating pvc artifact-passing-1-workdir" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.833Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.833Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.833Z" level=info msg="Steps node artifact-passing-1 initialized Running" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.833Z" level=info msg="StepGroup node artifact-passing-1-1968198367 initialized Running" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.833Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.834Z" level=info msg="Pod node artifact-passing-1-769410733 initialized Pending" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.853Z" level=info msg="Created pod: artifact-passing-1[0].generate-artifact (artifact-passing-1-hello-world-to-file-769410733)" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.853Z" level=info msg="Workflow step group node artifact-passing-1-1968198367 not yet completed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.853Z" level=info msg="TaskSet Reconciliation" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.853Z" level=info msg=reconcileAgentPod namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:11.853Z" level=info msg="Workflow to be dehydrated" Workflow Size=4825
time="2025-01-09T05:56:11.862Z" level=info msg="Workflow update successful" namespace=ns-1 phase=Running resourceVersion=46349684 workflow=artifact-passing-1
time="2025-01-09T05:56:21.824Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=46349684 namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:21.824Z" level=info msg="Task-result reconciliation" namespace=ns-1 numObjs=1 workflow=artifact-passing-1
time="2025-01-09T05:56:21.824Z" level=info msg="task-result changed" namespace=ns-1 nodeID=artifact-passing-1-769410733 workflow=artifact-passing-1
time="2025-01-09T05:56:21.825Z" level=info msg="node changed" namespace=ns-1 new.message= new.phase=Running new.progress=0/1 nodeID=artifact-passing-1-769410733 old.message= old.phase=Pending old.progress=0/1 workflow=artifact-passing-1
time="2025-01-09T05:56:21.826Z" level=info msg="Workflow step group node artifact-passing-1-1968198367 not yet completed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:21.826Z" level=info msg="TaskSet Reconciliation" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:21.826Z" level=info msg=reconcileAgentPod namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:21.826Z" level=info msg="Workflow to be dehydrated" Workflow Size=5289
time="2025-01-09T05:56:21.830Z" level=info msg="cleaning up pod" action=terminateContainers key=ns-1/artifact-passing-1-hello-world-to-file-769410733/terminateContainers
time="2025-01-09T05:56:21.830Z" level=info msg="https://10.0.0.1:443/api/v1/namespaces/ns-1/pods/artifact-passing-1-hello-world-to-file-769410733/exec?command=%2Fvar%2Frun%2Fargo%2Fargoexec&command=kill&command=15&command=1&container=wait&stderr=true&stdout=true&tty=false"
time="2025-01-09T05:56:21.836Z" level=info msg="Workflow update successful" namespace=ns-1 phase=Running resourceVersion=46349751 workflow=artifact-passing-1
time="2025-01-09T05:56:21.858Z" level=info msg="signaled container" container=wait error="unable to upgrade connection: container not found (\"wait\")" namespace=ns-1 pod=artifact-passing-1-hello-world-to-file-769410733 stderr="" stdout=""
time="2025-01-09T05:56:32.037Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=46349751 namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.038Z" level=info msg="Task-result reconciliation" namespace=ns-1 numObjs=1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.038Z" level=info msg="task-result changed" namespace=ns-1 nodeID=artifact-passing-1-769410733 workflow=artifact-passing-1
time="2025-01-09T05:56:32.038Z" level=info msg="Pod failed: Error (exit code 1): archive/tar: write too long" displayName=generate-artifact namespace=ns-1 pod=artifact-passing-1-hello-world-to-file-769410733 templateName=hello-world-to-file workflow=artifact-passing-1
time="2025-01-09T05:56:32.038Z" level=info msg="node changed" namespace=ns-1 new.message="Error (exit code 1): archive/tar: write too long" new.phase=Error new.progress=0/1 nodeID=artifact-passing-1-769410733 old.message= old.phase=Running old.progress=0/1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="Step group node artifact-passing-1-1968198367 deemed failed: child 'artifact-passing-1-769410733' failed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="node artifact-passing-1-1968198367 phase Running -> Failed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="node artifact-passing-1-1968198367 message: child 'artifact-passing-1-769410733' failed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="node artifact-passing-1-1968198367 finished: 2025-01-09 05:56:32.03975021 +0000 UTC" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="step group artifact-passing-1-1968198367 was unsuccessful: child 'artifact-passing-1-769410733' failed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="Outbound nodes of artifact-passing-1-769410733 is [artifact-passing-1-769410733]" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="Outbound nodes of artifact-passing-1 is [artifact-passing-1-769410733]" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="node artifact-passing-1 phase Running -> Failed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="node artifact-passing-1 message: child 'artifact-passing-1-769410733' failed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.039Z" level=info msg="node artifact-passing-1 finished: 2025-01-09 05:56:32.039922286 +0000 UTC" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.040Z" level=info msg="TaskSet Reconciliation" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.040Z" level=info msg=reconcileAgentPod namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.040Z" level=info msg="Updated phase Running -> Failed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.040Z" level=info msg="Updated message -> child 'artifact-passing-1-769410733' failed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.040Z" level=info msg="Marking workflow completed" namespace=ns-1 workflow=artifact-passing-1
time="2025-01-09T05:56:32.040Z" level=info msg="Workflow to be dehydrated" Workflow Size=5638
time="2025-01-09T05:56:32.046Z" level=info msg="cleaning up pod" action=deletePod key=ns-1/artifact-passing-1-1340600742-agent/deletePod
time="2025-01-09T05:56:32.051Z" level=info msg="Workflow update successful" namespace=ns-1 phase=Failed resourceVersion=46349807 workflow=artifact-passing-1
time="2025-01-09T05:56:32.064Z" level=info msg="cleaning up pod" action=labelPodCompleted key=ns-1/artifact-passing-1-hello-world-to-file-769410733/labelPodCompleted
time="2025-01-09T05:56:51.858Z" level=info msg="cleaning up pod" action=killContainers key=ns-1/artifact-passing-1-hello-world-to-file-769410733/killContainers
time="2025-01-09T05:58:23.777Z" level=info msg="Alloc=9688 TotalAlloc=4532919 Sys=48229 NumGC=2039 Goroutines=177"
```

### Logs from in your workflow's wait container

```text
time="2025-01-09T05:56:18.396Z" level=info msg="Starting Workflow Executor" version=v3.5.8
time="2025-01-09T05:56:18.404Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-01-09T05:56:18.404Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=ns-1 podName=artifact-passing-1-hello-world-to-file-769410733 templateName=hello-world-to-file version="&Version{Version:v3.5.8,BuildDate:2024-06-18T03:43:17Z,GitCommit:3bb637c0261f8c08d4346175bb8b1024719a1f11,GitTag:v3.5.8,GitTreeState:clean,GoVersion:go1.21.10,Compiler:gc,Platform:linux/amd64,}"
time="2025-01-09T05:56:18.420Z" level=info msg="Starting deadline monitor"
time="2025-01-09T05:56:20.421Z" level=info msg="Main container completed" error=""
time="2025-01-09T05:56:20.421Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-01-09T05:56:20.421Z" level=info msg="No output parameters"
time="2025-01-09T05:56:20.421Z" level=info msg="Saving output artifacts"
time="2025-01-09T05:56:20.422Z" level=info msg="Staging artifact: hello-art"
time="2025-01-09T05:56:20.422Z" level=info msg="Staging /workspace/model-detect from mirrored volume mount /mainctrfs/workspace/model-detect"
time="2025-01-09T05:56:20.422Z" level=info msg="Taring /mainctrfs/workspace/model-detect"
time="2025-01-09T05:56:21.429Z" level=info msg="archived 2 files/dirs in /mainctrfs/workspace/model-detect"
time="2025-01-09T05:56:21.430Z" level=error msg="executor error: archive/tar: write too long"
time="2025-01-09T05:56:21.430Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifact-passing-1/artifact-passing-1-hello-world-to-file-769410733/main.log"
time="2025-01-09T05:56:21.430Z" level=info msg="Creating minio client using static credentials" endpoint="10.127.188.91:32006"
time="2025-01-09T05:56:21.431Z" level=info msg="Saving file to s3" bucket=juicefs-minio-conf endpoint="10.127.188.91:32006" key=artifact-passing-1/artifact-passing-1-hello-world-to-file-769410733/main.log path=/tmp/argo/outputs/logs/main.log
time="2025-01-09T05:56:21.514Z" level=info msg="Save artifact" artifactName=main-logs duration=83.926139ms error="" key=artifact-passing-1/artifact-passing-1-hello-world-to-file-769410733/main.log
time="2025-01-09T05:56:21.514Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2025-01-09T05:56:21.514Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2025-01-09T05:56:21.526Z" level=info msg="Alloc=9933 TotalAlloc=16230 Sys=36965 NumGC=4 Goroutines=10"
time="2025-01-09T05:56:21.531Z" level=fatal msg="archive/tar: write too long"
```

Start by reproducing the failure with the provided workflow, focusing on the wait container and the generate-artifact template's /workspace/model-detect output. Trace the artifact archiving path responsible for the reported archive/tar error and determine whether an 800MB directory can complete successfully; done means the reproduction succeeds or the supported limitation and configuration are documented.

Written by the indexing model from the issue text.