oxidecomputer / oxidecomputer/crucible
downstairs should include span latencies in responses
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 260
- Forks
- 34
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 8
Description
It would be good for the downstairs to record several hrtime values in the process of servicing a job request, and then include them in the response message to the upstairs. This would enable us to apportion the time spent in various processing phases for every I/O. I think we would want times that roughly line up with when we would otherwise want DTrace probes anyway; e.g.,
- immediately after pulling bytes off the wire and deserialising them, store a time as the earliest time we've seen the job
- right before, and right after, making a system call (e.g., read() or write() or fsync() to the underlying data files)
- as we are serialising the response and are about to put it on the wire
If we do the same sort of thing in the upstairs, we can begin to infer how much time was spent in the kernel networking stack or on the wire between systems, or queued in a socket buffer, etc, separately from any queueing and from the I/O service time.
These numbers may also be interesting in the limit for heuristic fault detection or automated assessment of performance degradation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the downstairs job-request handler, response serialization, and read/write/fsync paths, then determine how the upstairs will consume the timing data. Done means each I/O response carries timestamps for the stated processing phases and the timing boundaries are agreed between upstairs and downstairs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, observability, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100