Fault injection

I've been reading about fuse, a mature circuit breaker library for Erlang (a platform known for "resiliency by default").

In circuit breakers configuration, they have two fuse types (you can think of them similar to toxics in Toxiproxy):

• Standard fuses, {standard, MaxR, MaxT}. These are fuses which tolerate MaxR melt attempts in a MaxT window, before they break down.
• Fault injection fuses, {fault_injection, Rate, MaxR, MaxT}. This fuse type sets up a fault injection scheme where the fuse fails at rate Rate, an floating point value between 0.0–1.0. If you enter, say 1 / 500 then roughly every 500th request will se a blown fuse, even if the fuse is okay. This can be used to add noise to the system and verify that calling systems support the failure modes appropriately. The values MaxR and MaxT works as in a standard fuse.

IMO, the idea of injecting faults through a circuit breaker is brilliant. Not every organization has adopted chaos engineering yet, but this could be a first step towards that, at least on the application level.

We should think about adopting this idea in Semian. The biggest concern would probably be development environment vs production: do we inject faults when it's running locally or on CI? If yes, how do we prevent test flakiness? Or should we do this only in production?

The issue names no files, tests, or entry points. Start by reviewing Semian’s existing circuit-breaker configuration and failure handling, then compare them with fuse’s standard and fault-injection types. A complete outcome would resolve whether fault injection belongs in development, CI, production, or a controlled combination, without introducing flaky tests.