crossplane / crossplane/upjet

TerraformPluginSDKAsyncConnector goroutine may share context with reconciliation goroutine

Open
#489 1 comment 3 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
481
Forks
131
Avg merge
2d 1h
Merged PRs (30d)
11

Description

### What happened?

If https://github.com/hashicorp/terraform-provider-google is used in a `terraform.SetupFn`, and the `context.Context` of given to the `terraform.SetupFn` is used as is, then it is likely the goroutine effectively performing the requests async rely on a context from the reconciliation loop goroutine. I believe this is the source of issues https://github.com/crossplane-contrib/provider-upjet-gcp/issues/538 or https://github.com/crossplane-contrib/provider-upjet-gcp/issues/611

- whichever context is given to `terraform.SetupFn` is **_stored_** in the meta field of `terraform.Setup` resulting from the configuration of the [terraform provider](https://github.com/hashicorp/terraform-provider-google/blob/d8d1db3267423af8b1fd1485bdbe22f5ff574e87/google/transport/config.go#L237)
- `terraform.SetupFn` is called with goroutine `Controller` Context to [synchronously to connect](https://github.com/crossplane/upjet/blob/ae37e28e15d96cb2630d7c666e8ceaff774e0a2f/pkg/controller/external_async_tfpluginfw.go#L55), ie during each reconciliation. The result is [stored to be reused](https://github.com/crossplane/upjet/blob/ae37e28e15d96cb2630d7c666e8ceaff774e0a2f/pkg/controller/external_tfpluginsdk.go#L114)
- the result `terraform.Setup` is later reused in asynchronous function [create](https://github.com/crossplane/upjet/blob/ae37e28e15d96cb2630d7c666e8ceaff774e0a2f/pkg/controller/external_async_tfpluginfw.go#L161) , [Update](https://github.com/crossplane/upjet/blob/ae37e28e15d96cb2630d7c666e8ceaff774e0a2f/pkg/controller/external_async_tfpluginfw.go#L194) or [Delete](https://github.com/crossplane/upjet/blob/ae37e28e15d96cb2630d7c666e8ceaff774e0a2f/pkg/controller/external_async_tfpluginfw.go#L227), with a total async timeout of one hour. When used, it relies on the stored context from goroutine `Controller` Context for all API calls to gcp.
- Currently, projects such as https://github.com/crossplane-contrib/provider-upjet-gcp need to strip any cancellation from the context otherwise [resources may never be created](https://github.com/crossplane-contrib/provider-upjet-gcp/issues/611)

### How can we reproduce it?

It is more likely to appear when the resource creation takes enough time for the reconciliation context to expire. [Which I believe is 3 minutes](https://github.com/crossplane/upjet/blob/ae37e28e15d96cb2630d7c666e8ceaff774e0a2f/pkg/pipeline/templates/controller.go.tmpl#L102). It has been reproduced with [gcp project operations](https://github.com/crossplane-contrib/provider-upjet-gcp/issues/611#issuecomment-2334093794)

This behavior can be reproduced with an example such as

```go
type example struct {
ctx context.Context
}

func (e *example) buildSetupFn() terraform.SetupFn {
return func(ctx context.Context, client client.Client, mg resource.Managed) (terraform.Setup, error) {
e.ctx = ctx
return terraform.Setup{
Meta: e,
}, nil
}
}

func (e *example) resource() *config.Resource {
return &config.Resource{
TerraformResource: &schema.Resource{
Create: func(rd *schema.ResourceData, i interface{}) error {
retrieved := i.(*example)
select {
case <-retrieved.ctx.Done():
return fmt.Errorf("done")
default:
return nil
}
},
// more fields...
}
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.