⛱️ Stripe Mutex Lock Contention
- Dominant language
- C++
- Stars
- 2k
- Forks
- 874
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 46
Description
# Summary
We have observed a lock contention issue with the Stripe mutex. This is an umbrella issue to track related changes.
# Problem
`Cache::open_read()` has severe lock contention - every read operation (includes cache lookup) acquires an exclusive lock on `stripe->mutex`, serializing all cache operations and limiting throughput.
https://github.com/apache/trafficserver/blob/7e366fa067f4916dbcec802eae049c0aea0acef6/src/iocore/cache/Cache.cc#L344
## Difficulties
I attempted to use a reader-writer lock instead of a mutex lock, and some proof-of-concept tests showed significant performance improvements. However, I found that we cannot simply replace this mutex lock with a reader-writer lock or a lock-free data structure. The main reason is that `StripeSM`, as a Continuation, requires the mutex lock when called from the event system.
https://github.com/apache/trafficserver/pull/12601 is recent another attempt by @bryancall.
### Event Handlers
- Event handlers of StripeSM
https://github.com/apache/trafficserver/blob/7e366fa067f4916dbcec802eae049c0aea0acef6/src/iocore/cache/StripeSM.h#L118-L122
### Dir operations
Some `Dir` functions seems read only operation, but it actually does write operation under some conditions.
- e.g. `Directory::probe()`
https://github.com/apache/trafficserver/blob/7e366fa067f4916dbcec802eae049c0aea0acef6/src/iocore/cache/CacheDir.cc#L528-L534
# Proposed Solution
Implement a two-tier locking architecture by decoupling `StripeSM` and `Stripe`:
1. Separate `StripeSM` (`Continuation`) and `Stripe` (shared data)
`StripeSM` (a Continuation) contains event handlers, while `Stripe` contains shared data.
Half of this change has already been completed by #11565 and related PRs, but we still need to clarify the separation between event handling and shared data access more explicitly.
2. Add Reader-Writer Lock to Stripe
Access to the shared data requires a reader-writer lock to allow concurrent reading. Alternatively, making Stripe a lock-free data structure (using RCU or Hazard Pointers) is another option.
3. Allocate StripeSM per Transaction
Each cache operation gets a lightweight StripeSM instance with its own mutex for event handling. It acquires an RW lock on the shared Stripe for data access.
## Architecture Diagram
Contributor guide
Assessment
This issue has not been assessed yet.