janestreet / janestreet/magic-trace
Omitting `--per-thread` leads to split traces (and other weird behavior)
- Dominant language
- OCaml
- Stars
- 6.3k
- Forks
- 209
- PR merge metrics
- No merged PRs in 30d
Description
As of #234 (when merged), `--per-thread` will be disabled sometimes even when not running with `-multi-thread`. However this means that a single snapshot can contain outdated data if a program switched CPUs. For example see the following traces:


The first image at the start zoomed in looks like this:

Both of these have large gaps in time between two parts of the trace because the data was stored in two separate buffers. Additionally the first image also has a decode error with pid of 0, and a cbr event with a unrealistic timestamp and incorrect pid.
Some potential changes could be made here. This issue is just to document this and lead to potential future investigation.
* magic-trace could always run with `--per-thread` if perf no longer fails to mmap extra events with this flag.
* magic-trace could detect that this single trace included multiple CPUs and instead only take the last contiguous chunk of data from the last CPU the process was running on when snapshotted. This would prevent the split traces even without `--per-thread`.
* magic-trace could filter the decode errors with pid of 0. It should be investigated why perf outputs these at all.
* magic-trace could try to detect invalid pids to eliminate that extraneous cbr event or figure out why perf gave this event at all.
Contributor guide
Research direction
Start by reproducing the split traces and decode anomalies described in the issue with and without --per-thread, then investigate how perf data is split across CPU buffers. Compare the proposed handling options and validate that the chosen approach prevents split traces and invalid events without introducing new failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ocaml
- Domain
- observability-sre, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100