oxidecomputer / oxidecomputer/omicron
internal hardening for availability
Open
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
There are some basic things we'll want to check everywhere (e.g., Nexus, Sled Agent, DNS servers, etc.) for availability:
- TCP KeepAlive: want to enable this on all network connections (in both directions) to identify failed systems. external vs. internal should probably have different values.
- HTTP KeepAlive: probably want to just pick a value like 60 seconds. Consider having clients make dummy requests to keep the connections open? (to avoid the problem of picking a connection that's been open for just under 60 seconds, sending a request, and having the server slam the door in your face -- we ran into this with Manta, admittedly only at very large scale since it's fairly improbable)
We'll want to review these, too. They might be more security-related (see #2184):
- limits for bad client behavior:
- maximum time waiting for a client to send request headers (whether on a new connection or between requests)
- minimum flow rate for request bodies (can be fairly low -- just want to avoid clients dribbling data in as a DoS vector to keep connections open)
- maximum number of open connections (ideally limited separately for different APIs -- e.g., external vs. internal)
- TCP listen socket backlog
- maximum rate of new connections created [ideally per-client]
- maximum rate of incoming requests [per authenticated user? or IP?, as well as overall]
- maximum number of connect-in-progress sockets
- maximum number of TLS-session-establishment-in-progress sockets
- size of tokio worker thread pool, blocked thread pool
- maximum length of time that graceful server shutdown can take
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the Nexus, Sled Agent, DNS, and server networking implementations, then scope the availability and bad-client protections listed here; done would require agreed behavior and coverage for the selected checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, networking, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100