argoproj / argoproj/argo-workflows
S3 client redownload entire dir when token expired
- Dominant language
- Go
- Stars
- 17k
- Forks
- 3.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 138
Description
### Pre-requisites
- [x] I have double-checked my configuration
- [x] I have tested with the `:latest` image tag (i.e. `quay.io/argoproj/workflow-controller:latest`) and can confirm the issue still exists on `:latest`. If not, I have explained why, **in detail**, in my description below.
- [x] I have searched existing issues and could not find a match for this bug
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/main/docs/CONTRIBUTING.md))
### What happened? What did you expect to happen?
When using node IAM or webidentity authentication,
s3 client obtains a temporary token with expiry time, (e.g. node IAM (6hr), webidentity (1hr))
We experience issue in the middle of downloading a folder, this token expires,
the retry process will restart downloading the entire folder again (then token expires again and repeat..)
We couldn't find an alternative way to address this besides changing the authentication method
We currently don't have a way to configure the request token expiration time
### Version(s)
v3.5.15
### Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
```YAML
N/A
```
### Logs from the workflow controller
```text
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
```
### Logs from in your workflow's wait container
```text
2025-11-10T01:43:22.239384732Z stderr F time="2025-11-10T01:43:22.239Z" level=info msg="Starting Workflow Executor" version=v3.5.14-atlan-1.3
2025-11-10T01:43:22.242797319Z stderr F time="2025-11-10T01:43:22.242Z" level=info msg="Using executor retry strategy" Duration=500ms Factor=1.1 Jitter=0.5 Steps=39
2025-11-10T01:43:22.242820809Z stderr F time="2025-11-10T01:43:22.242Z" level=info msg="Executor initialized" deadline="2025-11-12 18:30:04 +0000 UTC" includeScriptOutput=false namespace=default podName=1762713000-cleanup-personal-spaces-114904810 templateName=cleanup-personal-spaces version="&Version{Version:v3.5.14}"
2025-11-10T01:43:22.347859088Z stderr F time="2025-11-10T01:43:22.347Z" level=info msg="Start loading input artifacts..."
2025-11-10T01:43:22.347869425Z stderr F time="2025-11-10T01:43:22.347Z" level=info msg="Downloading artifact: metadata-extract"
2025-11-10T01:43:22.348013367Z stderr F time="2025-11-10T01:43:22.347Z" level=info msg="S3 Load path: /argo/inputs/artifacts/metadata-extract.tmp, key: argo-artifacts/default/1737957213/metadata-extract/1762713000/"
2025-11-10T01:43:22.355444197Z stderr F time="2025-11-10T01:43:22.355Z" level=info msg="Creating minio client using AWS SDK credentials"
2025-11-10T01:43:22.991910892Z stderr F time="2025-11-10T01:43:22.991Z" level=info msg="Getting file from s3" bucket=84918asd9js2 endpoint=[s3.eu-west-2.amazonaws.com](http://s3.eu-west-2.amazonaws.com/) key=argo-artifacts/default/1737957213/metadata-extract/1762713000/ path=/argo/inputs/artifacts/metadata-extract.tmp
2025-11-10T01:43:23.648494066Z stderr F time="2025-11-10T01:43:23.648Z" level=info msg="Getting directory from s3" bucket=84918asd9js2 endpoint=[s3.eu-west-2.amazonaws.com](http://s3.eu-west-2.amazonaws.com/) key=argo-artifacts/default/1737957213/metadata-extract/1762713000/ path=/argo/inputs/artifacts/metadata-extract.tmp
2025-11-10T01:43:23.648512558Z stderr F time="2025-11-10T01:43:23.648Z" level=info msg="Listing directory from s3" bucket=84918asd9js2 endpoint=[s3.eu-west-2.amazonaws.com](http://s3.eu-west-2.amazonaws.com/) key=argo-artifacts/default/1737957213/metadata-extract/1762713000/
2025-11-10T02:03:08.421748956Z stderr F time="2025-11-10T02:03:08.421Z" level=info msg="Starting Workflow Executor" version=v3.5.14-atlan-1.3
2025-11-10T02:03:08.425246742Z stderr F time="2025-11-10T02:03:08.425Z" level=info msg="Using executor retry strategy" Duration=500ms Factor=1.1 Jitter=0.5 Steps=39
2025-11-10T02:03:08.425261509Z stderr F time="2025-11-10T02:03:08.425Z" level=info msg="Executor initialized" deadline="2025-11-12 18:30:04 +0000 UTC" includeScriptOutput=false namespace=default podName=1762713000-cleanup-personal-spaces-114904810 templateName=cleanup-personal-spaces version="&Version{Version:v3.5.14-atlan-1.3,BuildDate:2025-10-14T01:27:09Z,GitCommit:unknown,GitTag:unknown,GitTreeState:unknown,GoVersion:go1.24.5,Compiler:gc,Platform:linux/amd64,}"
2025-11-10T02:03:08.526906703Z stderr F time="2025-11-10T02:03:08.526Z" level=info msg="Start loading input artifacts..."
2025-11-10T02:03:08.526927471Z stderr F time="2025-11-10T02:03:08.526Z" level=info msg="Downloading artifact: metadata-extract"
2025-11-10T02:03:08.526931489Z stderr F time="2025-11-10T02:03:08.526Z" level=info msg="S3 Load path: /argo/inputs/artifacts/metadata-extract.tmp, key: argo-artifacts/default/1737957213/metadata-extract/1762713000/"
2025-11-10T02:03:08.53190783Z stderr F time="2025-11-10T02:03:08.531Z" level=info msg="Creating minio client using AWS SDK credentials"
2025-11-10T02:03:09.168897942Z stderr F time="2025-11-10T02:03:09.168Z" level=info msg="Getting file from s3" bucket=84918asd9js2 endpoint=[s3.eu-west-2.amazonaws.com](http://s3.eu-west-2.amazonaws.com/) key=argo-artifacts/default/1737957213/metadata-extract/1762713000/ path=/argo/inputs/artifacts/metadata-extract.tmp
2025-11-10T02:03:09.81654035Z stderr F time="2025-11-10T02:03:09.816Z" level=info msg="Getting directory from s3" bucket=84918asd9js2 endpoint=[s3.eu-west-2.amazonaws.com](http://s3.eu-west-2.amazonaws.com/) key=argo-artifacts/default/1737957213/metadata-extract/1762713000/ path=/argo/inputs/artifacts/metadata-extract.tmp
2025-11-10T02:03:09.816559171Z stderr F time="2025-11-10T02:03:09.816Z" level=info msg="Listing directory from s3" bucket=84918asd9js2 endpoint=[s3.eu-west-2.amazonaws.com](http://s3.eu-west-2.amazonaws.com/) key=argo-artifacts/default/1737957213/metadata-extract/1762713000/
2025-11-10T03:03:19.775161246Z stderr F time="2025-11-10T03:03:19.774Z" level=info msg="Transient error: The provided token has expired."
2025-11-10T03:03:19.775183771Z stderr F time="2025-11-10T03:03:19.774Z" level=info msg="Transient error: The provided token has expired."
2025-11-10T03:03:20.413206995Z stderr F time="2025-11-10T03:03:20.412Z" level=info msg="S3 Load path: /argo/inputs/artifacts/metadata-extract.tmp, key: argo-artifacts/default/1737957213/metadata-extract/1762713000/"
2025-11-10T03:03:20.414941954Z stderr F time="2025-11-10T03:03:20.414Z" level=info msg="Creating minio client using AWS SDK credentials"
2025-11-10T03:03:21.013226066Z stderr F time="2025-11-10T03:03:21.013Z" level=info msg="Getting file from s3" bucket=84918asd9js2 endpoint=[s3.eu-west-2.amazonaws.com](http://s3.eu-west-2.amazonaws.com/) key=argo-artifacts/default/1737957213/metadata-extract/1762713000/ path=/argo/inputs/artifacts/metadata-extract.tmp
2025-11-10T03:03:21.013269095Z stderr F time="2025-11-10T03:03:21.013Z" level=warning msg="Non-transient error: fileName is a directory."
2025-11-10T03:03:21.013282133Z stderr F time="2025-11-10T03:03:21.013Z" level=warning msg="Non-transient error: fileName is a directory."
2025-11-10T03:03:21.013285252Z stderr F time="2025-11-10T03:03:21.013Z" level=info msg="Load artifact" artifactName=metadata-extract duration=1h0m12.486377049s error="failed to get file: fileName is a directory." key=argo-artifacts/default/1737957213/metadata-extract/1762713000/
2025-11-10T03:03:21.013288236Z stderr F time="2025-11-10T03:03:21.013Z" level=error msg="executor error: artifact metadata-extract failed to load: failed to get file: fileName is a directory."
2025-11-10T03:03:21.013445322Z stderr F time="2025-11-10T03:03:21.013Z" level=info msg="Alloc=17489 TotalAlloc=927150 Sys=36181 NumGC=109 Goroutines=6"
2025-11-10T03:03:21.013667696Z stderr F time="2025-11-10T03:03:21.013Z" level=fatal msg="artifact metadata-extract failed to load: failed to get file: fileName is a directory."
```
Contributor guide
Research direction
Start by tracing the S3 artifact-loading and retry path used for directory downloads, using the logged token-expiration failure as the reproduction scenario. Investigate how IAM or webidentity credentials are refreshed and how partial directory progress is handled; done means an expired token does not restart the entire download or fail with "fileName is a directory."
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 40/100