microsoft / microsoft/perfview

Aligned reading strategy in StreamReaderWriter causes event latency for EventPipe

Open
#1,447 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
4.7k
Forks
775
Avg merge
5d 11h
Merged PRs (30d)
9

Description

https://github.com/microsoft/perfview/blob/8a34d2d64bc958902b2fa8ea5799437df57d8de2/src/FastSerialization/StreamReaderWriter.cs#L791-L814

The above code in StreamReaderWriter can cause latency for EventPipe. If you have infrequent events, e.g., EventCounters, and an event block ends up a length not divisible by 8, then the reader will attempt a blocking read that will not return until more data is sent. In the case of infrequent data, this can create artificial latency.

For example, let's say there's an EventCounter session that is turned on with a frequency of 5 seconds. If any of the counter events in that stream end up in an event block that is not divisible by 8, then the reader will read the entire event block and attempt to read the remaining bytes to get to a mod(8) value. This causes EventPipeEventSource to not realize the event until the next counter shows up. This artificially adds 5 seconds of latency to potentially the entire stream depending on how frequently event blocks end up indivisible by 8.

I found this issue when triaging dotnet/diagnostics#2044. A bandaid fix is to turn on CPU sampling which will send a constant data stream guaranteeing that there will be more data to meet the reader's aligned reading requirements.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the aligned-reading code in src/FastSerialization/StreamReaderWriter.cs at lines 791-814, then trace how EventPipeEventSource consumes the stream. Reproduce the behavior with infrequent EventCounters and an event block whose length is not divisible by 8. Done means the reader no longer adds artificial latency while preserving correct stream processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.