oxidecomputer / oxidecomputer/crucible

downstairs should include span latencies in responses

Open
#1,282 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
260
Forks
34
Avg merge
2d 1h
Merged PRs (30d)
8

Description

It would be good for the downstairs to record several hrtime values in the process of servicing a job request, and then include them in the response message to the upstairs. This would enable us to apportion the time spent in various processing phases for every I/O. I think we would want times that roughly line up with when we would otherwise want DTrace probes anyway; e.g.,

  • immediately after pulling bytes off the wire and deserialising them, store a time as the earliest time we've seen the job
  • right before, and right after, making a system call (e.g., read() or write() or fsync() to the underlying data files)
  • as we are serialising the response and are about to put it on the wire

If we do the same sort of thing in the upstairs, we can begin to infer how much time was spent in the kernel networking stack or on the wire between systems, or queued in a socket buffer, etc, separately from any queueing and from the I/O service time.

These numbers may also be interesting in the limit for heuristic fault detection or automated assessment of performance degradation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by locating the downstairs job-request handler, response serialization, and read/write/fsync paths, then determine how the upstairs will consume the timing data. Done means each I/O response carries timestamps for the stated processing phases and the timing boundaries are agreed between upstairs and downstairs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, observability, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.