oxidecomputer / oxidecomputer/omicron

internal hardening for availability

Open
#2,414 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

There are some basic things we'll want to check everywhere (e.g., Nexus, Sled Agent, DNS servers, etc.) for availability:

  • TCP KeepAlive: want to enable this on all network connections (in both directions) to identify failed systems. external vs. internal should probably have different values.
  • HTTP KeepAlive: probably want to just pick a value like 60 seconds. Consider having clients make dummy requests to keep the connections open? (to avoid the problem of picking a connection that's been open for just under 60 seconds, sending a request, and having the server slam the door in your face -- we ran into this with Manta, admittedly only at very large scale since it's fairly improbable)

We'll want to review these, too. They might be more security-related (see #2184):

  • limits for bad client behavior:
    • maximum time waiting for a client to send request headers (whether on a new connection or between requests)
    • minimum flow rate for request bodies (can be fairly low -- just want to avoid clients dribbling data in as a DoS vector to keep connections open)
    • maximum number of open connections (ideally limited separately for different APIs -- e.g., external vs. internal)
    • TCP listen socket backlog
    • maximum rate of new connections created [ideally per-client]
    • maximum rate of incoming requests [per authenticated user? or IP?, as well as overall]
    • maximum number of connect-in-progress sockets
    • maximum number of TLS-session-establishment-in-progress sockets
  • size of tokio worker thread pool, blocked thread pool
  • maximum length of time that graceful server shutdown can take

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by locating the Nexus, Sled Agent, DNS, and server networking implementations, then scope the availability and bad-client protections listed here; done would require agreed behavior and coverage for the selected checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, networking, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.