crossplane / crossplane/uptest

Testing multiple resources linked with Usages lead to failure in test even though deletion succeeds

Open
#33 0 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Go
Stars
28
Forks
15
PR merge metrics
No merged PRs in 30d

Description

### What happened?

When testing combinations of resources that depend on each other with Usages, the delete step fails even though the cleanup eventually succeeds. The test contained following resources:

- XAWSLBController
- XEKS
- XNetwork

XAWSLBController contains a helm chart and a Usage of XEKS by a helm Release.

Running uptest against such a configuration leads to the following situation:

- We delete XAWSLBController first which succeeds immediately
- Afterwards we delete XEKS and get en error because cleanup in the background of XAWSLBController didn't finish yet, the Usage is still around

This is the error we're seeing:
```
"nousages.apiextensions.crossplane.io" denied the request: This resource is in-use by 1 Usage(s), including the Usage "configuration-aws-lb-controller-7d62z" by resource Release/configuration-aws-lb-controller-xmlng.
```

Working around this as of today is possible with a `pre-delete` or `post-delete` hook but we feel that it's not a great approach as it introduces additional hurdles for people trying uptest and it's leaking orchestration details which are already handled inside the core of crossplane.

Things we tried and discussed:

- omit "wait: true" from the delete statement. This has the same effect.
- run delete steps with `|| true`. This is possible and cleanup will succeed but we will swallow all errors regardless if it's connected to usages or not
- running in a loop and catch exit-code and stderr, then compare with stderr on failure with `"nousages.apiextensions.crossplane.io" denied the request`. Can work but can cause trouble with additional `pre-delete` and similar hooks as they now would require to be idempotent or we need to catch additional errors.
- finding all usages via owner-references connected to the current xr to delete and issue `kubectl wait` to wait for usages to be cleaned up before proceeding. Can also work but leaps deeply into internals.

Another thought: It might be that we're trying to work around a behavior which is actively not supported in chainsaw and this is why proposed solutions look kinda ugly. Maybe it's worthwhile creating a PR on chainsaw introducing something like retry paramter for a `script` step. At the very least we'll get some feedback from maintainers how they envision such a flow to work because trying to delete an object temporarily protected by an admission webhook might be not a standard usecase but i can imagine other setups can hit the same wall without being specific to crossplane.

Sidenote: This only occurs when the objects in question are NOT part of the same composition, as in this case deletion-errors are not visible to the outside and crossplane handles the cleanup flawlessly.

### How can we reproduce it?

Running uptest on this changeset without including the post-delete hook: https://github.com/upbound/configuration-aws-lb-controller/pull/1

### What environment did it happen in?

* Uptest Version: v1.1.2

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.