BrighterCommand / BrighterCommand/Brighter

Soak Tests

Open
#4,257 0 comments 0 reactions 0 assignees View on GitHub
.NET 0 - Backlog feature request Maintenance Performance Improvement V10.X
Dominant language
C#
Stars
2.5k
Forks
296
Avg merge
1d 11h
Merged PRs (30d)
21

Description

**Is your feature request related to a problem? Please describe.**

Brighter has recently experienced a memory leak during its transient lifetime for transformers/mappers. The presence of this leak is driven by our lack of soak testing for a release. Developer tests and sample runs give us release confidence, but don't catch issues like memory leaks as easily.

We should be able to find leaks before a release, particularly those affecting the factory pathways that enable us to create code using an underlying DI framework (specifically, the service collection).

This issue goes beyond #4252 and #4254, which address a reported leak, and instead asks, "How do we stop this in future?"

**Describe the solution you'd like**

A dedicated long-running console project that drives the real paths in a loop for N minutes/hours and samples memory over time:

- Drive Send/Publish through the DI-backed CommandProcessor and run ServiceActivator consuming from an in-memory/bus transport
- Sample retained memory correctly. The critical pitfall: force a full blocking GC before every sample and measure GC.GetGCMemoryInfo() / gen2 / LOH — otherwise, Server GC lazily retaining memory produces false-positive "climbing" curves. A true leak is a retained heap that rises monotonically across forced full collections, not working-set noise.
- Fail the run if the retained-heap regression slope over the window exceeds a threshold.

Tooling to reach for:
- dotnet-counters — live GC heap / gen2 / allocation-rate / working set; easy to log in CI.
- dotnet-gcdump — snapshot the heap at start and end and diff object counts; this is how you find which type is piling up, and it's collection-independent so it's not fooled by GC timing.
- dotnet-monitor sidecar — can auto-trigger a gcdump on a memory threshold if you run the soak in a container.

Persist the soak's retained-heap slope + a gcdump object-count baseline per release, and diff subsequent runs against it (store as a JSON baseline artifact, or use gh-action-benchmark-style trend tracking).

**Describe alternatives you've considered**

We do have a benchmark project, but this is focused on performance over soak testing.
K6 is a common driver for load/soak testing but is focused on web testing.

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the existing benchmark project and the DI-backed CommandProcessor and ServiceActivator paths described in the issue. Define a long-running console soak that exercises those paths, forces full blocking GCs before sampling, and records retained-heap slope and gcdump object-count baselines; done means threshold failures and release-comparable artifacts are produced.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, observability-sre, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.