☂️ Fix Flaky Tests

# Fix Flaky Tests

Consul has, by its nature as a large distributed system, a large set of non-determinstic tests. Whether a result of timeouts that are not configured to handle long waits on CI, or various other causes, sometimes tests fail randomly.

This contributes to a certain amount of noise when checking the results test suits locally or in CI, and increases the time it takes to develop and coordinate on new features. While eliminating _all_ non-deterministic test behavior would be ideal, the time-to-implement to benefit ratio isn't very compelling.

So, let's enumerate the tests that can flake! And, revisiting this list over time, research those issues that provide the highest value to fix ASAP vs. less-urgent tests.

Additionally, simple tests which require little consul-specific context can be a great launch point for new contributors to get involved in. Let's apply the `good first issue` label in those cases.

## By component
### Agent
**TestAgent**

| Done | Issue | Success rate % local | Notes |
| ----- | ----- | ------------------- | ----- |
| | #7046 | 99.9% ish | todo(kit): test under resource contention |
| | #7051 | 100% | todo(kit): test under resource contention |
| | #7056 | 100% | todo(kit): test under resource contention |
| | #7059 | 94.3% | |
| | #7066 | 100% | todo(kit): test under resource contention |
| | #7110 | 94.4% | |
| ✅ | #7119 | | Could not repro |
| ✅ | #7145 | | Fixed w/ WaitForActiveCARoot |
| | #7667 | | Needs profiling |

#### **agent/consul TestAutoEncryptSign**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7133 | 100% ?? | Kit: only failed localwhen running package tests... needs more profiling |

#### **agent/consul TestAutoEncryptSign**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7133 | 100% ?? | Kit: only failed localwhen running package tests... needs more profiling |

#### **agent/consul TestConnectCAConfig**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7132 | 100% ?? | Kit: only failed localwhen running package tests... needs more profiling |
| #7131 | 100% ?? | Kit: only failed localwhen running package tests... needs more profiling |

**TestACL**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7063 | | Needs to be profiled |
| #7052 | | Needs to be profiled |

**TestAutopilot**

| Issue | Success rate % local | Notes |
| ----- | ------------------- | ----- |
| #7084 | | Slow, hard to get a large enough sample to profile |

**TestCache**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7045 | | Needs to be profiled |
| #7050 | | Needs to be profiled |

**TestCatalog**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7049 | | Needs to be profiled |
| #7065 | | Needs to be profiled |
| #7666 | | Needs to be profiled |

**TestConnect**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7044 | | Needs to be profiled |
| #7067 | | Needs to be profiled |
| #7085 | 96% | |

**TestSnapshot**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7054 | | Needs to be profiled |
| #7064 | | Needs to be profiled |

**TestAPI**

| Done? | Issue | Success rate % local | Notes |
| ------ | ----- | --------------------- | ----- |
| | #7060 | | Needs to be profiled |
| | #7064 | | Needs to be profiled |
| ✅ | #7053 | | Needs to be profiled |

**TestDNS**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7068 | 100% | Kit: I could not repro a failure locally by running it a lot, try using `make test-flake` to see it under resource constraints |

**TestGenerate**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7062 | | Needs to be profiled |

**TestHealth**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7055 | | Needs to be profiled |

**TestLeader**

| Done | Issue | Success rate % local | Notes |
| ----- | ----- | --------------------- | ----- |
| 📑 | #7043 | 70% | Broke with recent change, the fact that it passed at all is a bit concerning|

**TestPProfHandlers**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7061 | | Needs to be profiled |

**TestPrepared**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7058 | | Needs to be profiled |

**TestTxn**

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7057 | 100% | Kit: Likely only fails in CI / under high resource contention |

### Misc.

| Issue | Success rate % local | Notes |
| ----- | --------------------- | ----- |
| #7048 | | Not a test but a sync tool, might help resolve issues caused by using a not-ready agent |
| #7047 | | Linter check in CI to ensure we clean up tests |

#### TODOs
- [x] Tables
- [ ] Add some addtl. categorization for the tests by {complexity, priority}?

_This is an umbrella issue. Umbrellas do not have their own implementation, but rather are a way to link together various related issues and their implementations under a common goal. Github automatically cross-references issues and PRs when mentioned, so adding an issue to an umbrella provides an easy way to loosely associate similar but independent tasks. They also provide a location for meta-level discussion of a goal that is larger than a single task. Umbrellas can be kept open for long-running living documentation for ongoing goals, or closed out and held for reference later if the goal has a clear stopping point._

This is an umbrella issue with no implementation of its own. Start by choosing one of the listed child issues, then run the relevant test or `make test-flake` as suggested in the table. Done means resolving that child issue's flaky behavior and updating its status here.

Written by the indexing model from the issue text.