janestreet / janestreet/magic-trace

Omitting `--per-thread` leads to split traces (and other weird behavior)

Open
#236 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
OCaml
Stars
6.3k
Forks
209
PR merge metrics
No merged PRs in 30d

Description

As of #234 (when merged), `--per-thread` will be disabled sometimes even when not running with `-multi-thread`. However this means that a single snapshot can contain outdated data if a program switched CPUs. For example see the following traces:

![image](https://user-images.githubusercontent.com/15175891/176778554-2ac3c437-0e88-4c14-a252-ebb10f7a7798.png)
![image](https://user-images.githubusercontent.com/15175891/176778629-88e6857d-4c9e-422a-bb7a-5fdc996e207d.png)

The first image at the start zoomed in looks like this:

![image](https://user-images.githubusercontent.com/15175891/176778699-f0eb7108-530e-4d18-b054-2d11788f0528.png)

Both of these have large gaps in time between two parts of the trace because the data was stored in two separate buffers. Additionally the first image also has a decode error with pid of 0, and a cbr event with a unrealistic timestamp and incorrect pid.

Some potential changes could be made here. This issue is just to document this and lead to potential future investigation.

* magic-trace could always run with `--per-thread` if perf no longer fails to mmap extra events with this flag.
* magic-trace could detect that this single trace included multiple CPUs and instead only take the last contiguous chunk of data from the last CPU the process was running on when snapshotted. This would prevent the split traces even without `--per-thread`.
* magic-trace could filter the decode errors with pid of 0. It should be investigated why perf outputs these at all.
* magic-trace could try to detect invalid pids to eliminate that extraneous cbr event or figure out why perf gave this event at all.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the split traces and decode anomalies described in the issue with and without --per-thread, then investigate how perf data is split across CPU buffers. Compare the proposed handling options and validate that the chosen approach prevents split traces and invalid events without introducing new failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
ocaml
Domain
observability-sre, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.