argoproj / argoproj/argo-workflows

artifactGC not working with IRSA

Open
#9,746 21 comments 0 reactions 0 assignees View on GitHub
area/artifacts area/gc hacktoberfest
Dominant language
Go
Stars
17k
Forks
3.7k
Avg merge
1d 15h
Merged PRs (30d)
138

Description

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

**what happened?**
artifactgc step failed, object still on s3 and workflow did not get archived

**what expected?**
artifactgc step succeed, object removed off s3 and workflow gets archived

### Version

v3.4.1

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
spec:
serviceAccountName: argo-workflows-server
artifactGC:
strategy: OnWorkflowCompletion
serviceAccountName: argo-workflows-workflow-controller
podMetadata:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::redact:role/redact..approle"
```

logs from the artgc pod:
```
{"log":"time=\"2022-10-04T10:16:46.328Z\" level=info msg=\"S3 Delete artifact: key: redact/main.log\"\n","stream":"stderr","time":"2022-10-04T10:16:46.32863739Z"}
{"log":"time=\"2022-10-04T10:16:46.328Z\" level=info msg=\"Creating minio client using AWS SDK credentials\"\n","stream":"stderr","time":"2022-10-04T10:16:46.328647872Z"}
{"log":"time=\"2022-10-04T10:16:46.573Z\" level=warning msg=\"Non-transient error: NoCredentialProviders: no valid providers in chain. Deprecated.\\n\\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors\"\n","stream":"stderr","time":"2022-10-04T10:16:46.573355156Z"}
```

note: for other AWS things to work i set volume/volumemounts/env variables as in https://github.com/argoproj/argo-workflows/discussions/7461 but artifactgc pod doesn't seem to accept that?

### Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

### Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Contributor guide

Open the contributing guide

Research direction

Reproduce the artifactGC workflow from the issue on v3.4.1, using the shown service accounts and IRSA pod annotation. Start by inspecting the artifactGC pod configuration and its logs for AWS credential propagation, then compare the workflow-controller and wait-container behavior. Done means artifactGC obtains credentials, removes the S3 object, and archives the workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.