Testing Improvement Plan

* CLI/API
* Coverage
* [ ] Merge #6075 (maybe with a few tweaks) and then fill out the flags that aren't covered.
* Language Interop
* [x] Fix the current interop tests (https://github.com/ipfs/interop/issues/71).
* [ ] Run sharness tests gainst both go-ipfs and js-ipfs (where possible).
* Backwards Compatibility
* [ ] Run sharness tests with previous go-ipfs binary as a client. Need to mark the required client version on sharness tests.
* [x] Interop tests: These may be sufficient.
* Performance
* [ ] Benchmarks in CI.
* [ ] End-to-end bitswap (#6446)
* [ ] Garbage Collection
* [ ] File Add
* [ ] File Cat/Get
* [ ] Directory Listing
* [ ] Pin ls/add/etc.
* [ ] Dial Latency (testing the dialer, transports, etc.).
* [ ] Stream throughput (using `ipfs p2p`).
* [ ] Performance history (https://github.com/ipfs/go-ipfs/issues/6483#issuecomment-508893503)
* Reliability/Stability
* [x] Nightly _Mirror_ Gateway: All requests to a specific gateway get mirrored to this gateway with nginx.
* [ ] Track and compare errors/latencies versus the gateway serving the response.
* [ ] Auto-file Issues.
* [ ] Auto deploy a nightly build.
* [ ] Auto-file issues on crash.
* [ ] Automatically pull goroutine dumps, cpu profiles, and memory profiles.
* [ ] Automatically analyze with [stackparse](https://github.com/ipfs/interop/issues/71) and auto-file issues when a specific goroutine count increases by 100x (usually indicates an issues). Note: this'll have a bunch of false positives but it's a good signal.
* [ ] Same with the heap: File an issue if the "top memory user in a steady state" changes.
* [ ] Same with the CPU profile.
* [ ] Nightly Bootstrapper
* [ ] Auto deploy a nightly build.
* [ ] Auto-file issues on crash.
* [ ] Automatically pull goroutine dumps, cpu profiles, and memory profiles.
* [ ] Automatically analyze with [stackparse](https://github.com/ipfs/interop/issues/71) and auto-file issues when a specific goroutine count increases by 100x (usually indicates an issues). Note: this'll have a bunch of false positives but it's a good signal.
* [ ] Same with the heap: File an issue if the "top memory user in a steady state" changes.
* [ ] Same with the CPU profile.
* Network Simulation: tests run on a simulated network so we can predict how are changes will affect the network. We should be able to run each test with X% of the network running the current release and the old release.
* [ ] DHT Tests - test node run as a client and server.
* [ ] FindPeer
* [ ] Provide/FindProviders
* [ ] IPNS
* [ ] End-to-end Bitswap (including finding providers, multiple providers, etc.).
* Network Canary - Continuous background tests against the main network.
* [ ] DHT Tests - test node run as a client and server.
* [ ] FindPeer
* [x] Provide/FindProviders
* [ ] IPNS
* [ ] End-to-end Bitswap (including finding providers, multiple providers, etc.).
* [ ] Gateway Test - Add a file to the test node and fetch it on the ipfs.io gateway (measuring latency).
* Release Candidates
* [ ] Opt-out telemetry is built into RC binaries for collection of network operation and performance (required to be able to confirm any hypothesises formed, from testlab tests, in the production network)

In both network tests, we should track and compare metrics on:

* Number of dials.
* Total network bandwidth.
* Latencies/times for each step.
* CPU Usage
* Memory Usage
* Goroutine Counts.

---

The Plan: https://docs.google.com/spreadsheets/d/1xyqyGUF-oe3x9ln88YonVeOMWWdknik74lVgL_3dBY8