apache / apache/trafficserver

⛱️ Stripe Mutex Lock Contention

Open
#12,788 0 comments 0 reactions 1 assignee Claimed by @masaori335 View on GitHub
Cache Performance
Dominant language
C++
Stars
2k
Forks
874
Avg merge
6d 15h
Merged PRs (30d)
46

Description

# Summary

We have observed a lock contention issue with the Stripe mutex. This is an umbrella issue to track related changes.

# Problem

`Cache::open_read()` has severe lock contention - every read operation (includes cache lookup) acquires an exclusive lock on `stripe->mutex`, serializing all cache operations and limiting throughput.

https://github.com/apache/trafficserver/blob/7e366fa067f4916dbcec802eae049c0aea0acef6/src/iocore/cache/Cache.cc#L344

## Difficulties

I attempted to use a reader-writer lock instead of a mutex lock, and some proof-of-concept tests showed significant performance improvements. However, I found that we cannot simply replace this mutex lock with a reader-writer lock or a lock-free data structure. The main reason is that `StripeSM`, as a Continuation, requires the mutex lock when called from the event system.

https://github.com/apache/trafficserver/pull/12601 is recent another attempt by @bryancall.

### Event Handlers

- Event handlers of StripeSM

https://github.com/apache/trafficserver/blob/7e366fa067f4916dbcec802eae049c0aea0acef6/src/iocore/cache/StripeSM.h#L118-L122

### Dir operations

Some `Dir` functions seems read only operation, but it actually does write operation under some conditions.

- e.g. `Directory::probe()`

https://github.com/apache/trafficserver/blob/7e366fa067f4916dbcec802eae049c0aea0acef6/src/iocore/cache/CacheDir.cc#L528-L534

# Proposed Solution

Implement a two-tier locking architecture by decoupling `StripeSM` and `Stripe`:

1. Separate `StripeSM` (`Continuation`) and `Stripe` (shared data)

`StripeSM` (a Continuation) contains event handlers, while `Stripe` contains shared data.

Half of this change has already been completed by #11565 and related PRs, but we still need to clarify the separation between event handling and shared data access more explicitly.

2. Add Reader-Writer Lock to Stripe

Access to the shared data requires a reader-writer lock to allow concurrent reading. Alternatively, making Stripe a lock-free data structure (using RCU or Hazard Pointers) is another option.

3. Allocate StripeSM per Transaction

Each cache operation gets a lightweight StripeSM instance with its own mutex for event handling. It acquires an RW lock on the shared Stripe for data access.

## Architecture Diagram

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.