TCP flow state tracking should be lockless, if possible

Open
#837 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Needs clarification
Activity status
Stale
Tech stack
rust

Research direction

Start by locating the fastpath TCP flow state update guarded by KMutex and trace how Rx and Tx packet processing reach it. Review TcpState transitions and counter updates, then determine whether the proposed compare-and-swap approach preserves correctness under concurrent access. Done means TCP state remains correct without per-flow mutex contention and the fastpath performance improves.

Written by the indexing model from the issue text.

Description

perf

For TCP flows, once a fastpath entry has been acquired we need to update the TCP state based on the flags seen in a packet. Today this is guarded using a KMutex<T>, which ends up as a place where contention can arise for a single flow. Mainly, this is because the Rx packet processing and Tx packet processing are going to be happening on different threads unless we get very lucky with squeue bindings.

Outright removing the TCP state gets us around 500 Mbps, but I think in general we do want to keep it in place. My thinking here is that most state transitions are going to be trivial: e.g., moving from TcpState::Established to itself, and incrememting some counters. Everything else could be probably handled by computing the transform, performing a compare-and-swap between the old and target states, and then looping those steps if the comparison fails.

Dominant language
Rust
Stars
77
Forks
11
Avg merge
9d 20h
Merged PRs (30d)
8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from oxidecomputer/opte

All issues in oxidecomputer/opte

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.