Using the unit tests as the PGO task has problems

Feature or enhancement

Proposal:

When Python is compiled with --enabled-optimizations, which turns on PGO (program guided optimizations), the build will run a subset of the unit tests as the "task" to generate profile information. Included in the profile is information like counts of the taken side of a CPU conditional branch instructions. To get the best optimization, your PGO task should match the branch taken behavior of your real workloads.

Using the unit tests has the advantage that we have good code coverage in terms of executing most branches and code paths. It also has the advantage of being available without any external dependencies. It has the disadvantage that the code executed during unit tests is likely quite atypical of what's executed during real applications. Running the ./python -X perf -m test --pgo under the "perf" tool, I see the following results:

This profile reveals a number of problems. First, a large fraction of time in spent in the cyclic GC. That's because the unit test framework calls test.support.gc_collect() before each test case. That function triggers three full GC collections. Other tests also call the GC explicitly. This is not behavior typical of a real program.

Also taking a lot of time are functions related to parsing and compiling Python code. Notice the builtin_eval() function, for example. I suspect that's mostly a result of using "doctest". Again, this would not be typical of real programs.

I think we should replace the PGO task with a program that more closely represents the behavior of real Python programs. There are at least two potential advantages: it could make the compiled Python binary faster for real programs, it could make our benchmark results less noisy since the compiler would be doing a better and most consistent job of generating optimal code.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-130702

首先检查当前的 --enable-optimizations PGO 任务和 ./python -X perf -m test --pgo 命令，包括 Lib/test/support/init.py 中的 test.support.gc_collect，以及涉及 Lib/json/encoder.py 的性能分析观察结果。完成的标准是定义并验证一个具有代表性的替代工作负载，然后将其性能分析结果和基准测试影响与现有的单元测试任务进行比较。