Finish uploading artifacts even when activeDeadlineSeconds is reached

# Summary

What change needs making?

Artifacts uploading should continue even when activeDeadlineSeconds is reached.

# Use Cases

When would you use this?

Users would like to see any intermediate results before time out for debugging and analysis.

## Example

Assuming you set up S3 upload and service account properly, you can submit this and it will not upload. Then if you delete && sleep 60 and submit again, it will work properly because it does not time out. (Or remove activeDeadlineSeconds)

```
metadata:
name: simple-timeout-artifact-repro
namespace: production
spec:
templates:
- name: main
outputs:
artifacts:
- name: test
path: /opt/test.txt
s3:
endpoint: s3.amazonaws.com
bucket: xxx
accessKeySecret:
name: xxx
key: ARGO_ARTIFACTS_ACCESS_KEY_ID
secretKeySecret:
name: xxx
key: ARGO_ARTIFACTS_ACCESS_KEY_SECRET
key: simple-timeout-artifact-repro/test.txt
archive:
none: {}
optional: true
container:
name: main
image: ubuntu:20.04
command:
- bash
- '-c'
args:
- echo "testing" && echo "testing" > /opt/test.txt && ls -al /opt && sleep 60
archiveLocation:
archiveLogs: true
activeDeadlineSeconds: 20
entrypoint: main
serviceAccountName: argo-test-runner
ttlStrategy:
secondsAfterCompletion: 30
secondsAfterSuccess: 30
secondsAfterFailure: 30
workflowMetadata:
labels:
workflows.argoproj.io/archive-strategy: always
```

wait container log:

```
time="2023-05-24T01:11:49 UTC" level=info msg="Starting deadline monitor"
time="2023-05-24T01:12:08 UTC" level=info msg="Deadline monitor stopped"
time="2023-05-24T01:12:08 UTC" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2023-05-24T01:12:08 UTC" level=warning msg="Non-transient error: context canceled"
time="2023-05-24T01:12:08 UTC" level=info msg="Main container completed" error="context canceled"
time="2023-05-24T01:12:08 UTC" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2023-05-24T01:12:08 UTC" level=info msg="No output parameters"
time="2023-05-24T01:12:08 UTC" level=info msg="Saving output artifacts"
time="2023-05-24T01:12:08 UTC" level=info msg="Staging artifact: test"
time="2023-05-24T01:12:08 UTC" level=info msg="Copying /opt/test.txt from container base image layer to /tmp/argo/outputs/artifacts/test.tg
z"
time="2023-05-24T01:12:08 UTC" level=info msg="/var/run/argo/outputs/artifacts/opt/test.txt.tgz -> /tmp/argo/outputs/artifacts/test.tgz"
time="2023-05-24T01:12:08 UTC" level=warning msg="Ignoring optional artifact 'test' which does not exist in path '/opt/test.txt': open /var
/run/argo/outputs/artifacts/opt/test.txt.tgz: no such file or directory"
time="2023-05-24T01:12:08 UTC" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: production/2023/05/24/simple-timeout-art
ifact-repro/simple-timeout-artifact-repro/main.log"
time="2023-05-24T01:12:08 UTC" level=info msg="Creating minio client using static credentials" endpoint=s3.amazonaws.com
time="2023-05-24T01:12:08 UTC" level=info msg="Saving file to s3" bucket=qawolf-argo-workflows endpoint=s3.amazonaws.com key=production/202
3/05/24/simple-timeout-artifact-repro/simple-timeout-artifact-repro/main.log path=/tmp/argo/outputs/logs/main.log
time="2023-05-24T01:12:08 UTC" level=info msg="Save artifact" artifactName=main-logs duration=503.699475ms error="" key=production/202
3/05/24/simple-timeout-artifact-repro/simple-timeout-artifact-repro/main.log
time="2023-05-24T01:12:08 UTC" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2023-05-24T01:12:08 UTC" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2023-05-24T01:12:08 UTC" level=info msg="Create workflowtaskresults 403"
time="2023-05-24T01:12:08 UTC" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj/
.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:production:argo
-test-runner\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"production\": RBAC: clusterro
le.rbac.authorization.k8s.io \"argo-cache-writer-production\" not found"
time="2023-05-24T01:12:08 UTC" level=info msg="Patch pods 200"
time="2023-05-24T01:12:08 UTC" level=info msg="Alloc=7853 TotalAlloc=16787 Sys=24018 NumGC=5 Goroutines=9"
```

These say the file does not exist, but the main container logs show it there from the `ls`.

Controller log:

```
time="2023-05-24T01:26:46.713Z" level=info msg="Processing workflow" namespace=dev workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="Task-result reconciliation" namespace=dev numObjs=0 workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="Pod failed: Pod was active on the node longer than the specified deadline" displayName=simple-time │
│ out-artifact-repro2 namespace=dev pod=simple-timeout-artifact-repro2 templateName=main workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=warning msg="workflow uses legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow │
│ -rbac/" namespace=dev workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="node changed" namespace=dev new.message="Pod was active on the node longer than the specified dead │
│ line" new.phase=Failed new.progress=0/1 nodeID=simple-timeout-artifact-repro2 old.message= old.phase=Running old.progress=0/1 workflow=simple-time │
│ out-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="TaskSet Reconciliation" namespace=dev workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg=reconcileAgentPod namespace=dev workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="Updated phase Running -> Failed" namespace=dev workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="Updated message -> Pod was active on the node longer than the specified deadline" namespace=dev w │
│ orkflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="Marking workflow completed" namespace=dev workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="Marking workflow as pending archiving" namespace=dev workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="Checking daemoned children of " namespace=dev workflow=simple-timeout-artifact-repro2 │
│ time="2023-05-24T01:26:46.713Z" level=info msg="Workflow to be dehydrated" Workflow Size=2025 │
│ time="2023-05-24T01:26:46.716Z" level=info msg="Create events 201" │
│ time="2023-05-24T01:26:46.718Z" level=info msg="Update workflows 200" │
│ time="2023-05-24T01:26:46.718Z" level=info msg="Workflow update successful" namespace=dev phase=Failed resourceVersion=287336 workflow=simple-time │
│ out-artifact-repro2 │
│ time="2023-05-24T01:26:46.719Z" level=info msg="cleaning up pod" action=deletePod key=dev/simple-timeout-artifact-repro2-1340600742-agent/deletePo │
│ d
```

---

**Message from the maintainers**:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

The report centers on Kubernetes activeDeadlineSeconds, the deadline monitor, artifact saving, and controller reconciliation. Start by tracing how deadline cancellation reaches artifact collection and compare the pod and controller logs shown here. Done means output artifacts remain uploadable after the deadline, with a regression test covering the provided reproduction.

Written by the indexing model from the issue text.