Improving perf for large power rail traces.
- Dominant language
- C++
- Stars
- 6.5k
- Forks
- 872
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 218
Description
I am looking to load large synthetic power rail traces into trace processor & perfetto but have noticed some large perf issues, and would like to investigate optimization.
I see 4 main areas that would likely need addressing when testing a trace with ~48 million power rail samples:
- trace_processor parse/load time
- UI Plugin on trace load time.
- queries run when expanded the power rails track group
- Aggregation queries when clicking and dragging over samples
This metatrace illustrates some of the bottlenecks in play for the later 3:
[metatrace.pftrace.zip](https://github.com/user-attachments/files/19806131/metatrace.pftrace.zip)
## Trace parse time.
Trace parse time seems to be quite slow for large traces of ~48 million samples. after running a few experiments it looks like the parse time does not scale linearly with sample count and instead much worse.
See the following experiments performed on a M1 pro
| rails | samples per rail | trace_processor_shell load time (seconds)|
| -------- | ------- | ------ |
| 48 | 100k | 5 |
| 48 | 500k | 66 |
| 48 | 1 million | 246 |
After some investigations it appears as if the ArgTracker used by power rails is the source of most of this non-linear performance scaling. As an experiment, if i remove [the lines of code responsible for that](https://github.com/google/perfetto/blob/2a02390b90b05850c5e7f81e530e878be8ff35f6/src/trace_processor/importers/proto/android_probes_parser.cc#L167-L171), I see the following instead, wich is roughly linear scaling.
| rails | samples per rail | trace_processor_shell load time (seconds)|
| -------- | ------- | ------ |
| 48 | 100k | 3 |
| 48 | 500k | 16 |
| 48 | 1 million | 32 |
Removing this arg would result in significantly improved performance, although i'm unaware of the consequences of doing so.
## Plugin on trace load time.
it appears we are spending a lot of time intializing the dev.perfetto.TrackEvent plugin here:https://github.com/google/perfetto/blob/2a02390b90b05850c5e7f81e530e878be8ff35f6/ui/src/plugins/dev.perfetto.TrackEvent/index.ts#L37-L59
through a chain of SQL module includes, this query seems to be the cause of the vast majority of plugin load time: https://github.com/google/perfetto/blob/2a02390b90b05850c5e7f81e530e878be8ff35f6/src/trace_processor/perfetto_sql/stdlib/viz/summary/track_event.sql#L41-L54
This makes sense given the sheer size of the counters table. it is unclear if there is a way to optimize that further.
## track loading
I don't have a firm grasp on how these queries operate, but i currently see 1 query per track in the meta trace:
if it is possible to merge these, we may be able to reduce the number of times the counter table needs to be scanned.
## track aggregation
the main bottleneck i'm seeing here is that we are running two aggregation queries desipite only the results of one being visible to the user at any point in time:
- The generic counter aggregation query.
- the power aggregation query which runs ~3x faster for power tracks than the generic aggregation query
two potential optimization solutions that would result in a ~3x improvement vs current perf:
- Only display the power aggregation query for these tracks and disable the generic aggregation query
- Be lazy about running aggregation queries, and don't run them until the tab is made visible to the user.
Contributor guide
Research direction
Reproduce the large power-rail trace timings with trace_processor_shell, then inspect ArgTracker in src/trace_processor/importers/proto/android_probes_parser.cc. Profile the TrackEvent plugin in ui/src/plugins/dev.perfetto.TrackEvent/index.ts and the summary query in src/trace_processor/perfetto_sql/stdlib/viz/summary/track_event.sql. Done would require a measured improvement across parsing, track loading, and aggregation without breaking power-rail behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql, typescript
- Domain
- data, frontend, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100