hashicorp / hashicorp/terraform-plugin-sdk

docs: Document best practices for parallel testing

Open
#1 0 comments 6 reactions 0 assignees View on GitHub
documentation subsystem/tests
Dominant language
Go
Stars
485
Forks
244
Avg merge
19h 57m
Merged PRs (30d)
4

Description

Historically the testing framework didn't have any way to run acceptance tests in parallel. We used a [separate tool](https://github.com/jen20/teamcity-go-test) to achieve this and we still use it today. This worked & works in context of official providers (`github.com/terraform-providers/*`), but not in others and most importantly not on developer's workstations.

This is one of the reasons @bflad introduced `ParallelTest ` in https://github.com/hashicorp/terraform/pull/18688

The goal is to document this with some examples from provider which already use this helper function.

There are clear benefits we can gain by parallel testing:

- It provides a way of ensuring the provider implementation & SDK & API works correctly under real load without having to wait until end users report misbehaviours to us.
- Terraform defaults to `parallelism=10` which means that it will issue up to 10 requests in parallel _by default_. Parallel testing therefore reflects how users would use the provider in the wild.
- This saves users & machines costly time and is one of essential features of Terraform.

There are however some (side) effects of this:
- Many APIs aren't ready to process more than 1 (or N) requests at the same time, without documenting this, or even knowing this. Maintainers often learn this hard-way through duplicate IDs returned for different resources, HTTP 500 responses, or otherwise misbehaving APIs.
- This then leads to maintainers reaching to API vendors to fix their APIs, and/or implement various workarounds in providers, e.g. [`mutexkv`](https://github.com/hashicorp/terraform/tree/6b6b5a43c32f7e6bc4b90db79d44494c72398819/helper/mutexkv) to practically prevent Terraform from sending parallel requests to the affected API.
- Many APIs implement some form of rate limiting, to prevent abuse.
- Many APIs have shared namespaces which expect unique resource names. Provider developers may use static names in tests, which then makes tests to fail as a result of name clash.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.