Using the unit tests as the PGO task has problems

Feature or enhancement

Proposal:

When Python is compiled with --enabled-optimizations, which turns on PGO (program guided optimizations), the build will run a subset of the unit tests as the "task" to generate profile information. Included in the profile is information like counts of the taken side of a CPU conditional branch instructions. To get the best optimization, your PGO task should match the branch taken behavior of your real workloads.

Using the unit tests has the advantage that we have good code coverage in terms of executing most branches and code paths. It also has the advantage of being available without any external dependencies. It has the disadvantage that the code executed during unit tests is likely quite atypical of what's executed during real applications. Running the ./python -X perf -m test --pgo under the "perf" tool, I see the following results:

This profile reveals a number of problems. First, a large fraction of time in spent in the cyclic GC. That's because the unit test framework calls test.support.gc_collect() before each test case. That function triggers three full GC collections. Other tests also call the GC explicitly. This is not behavior typical of a real program.

Also taking a lot of time are functions related to parsing and compiling Python code. Notice the builtin_eval() function, for example. I suspect that's mostly a result of using "doctest". Again, this would not be typical of real programs.

I think we should replace the PGO task with a program that more closely represents the behavior of real Python programs. There are at least two potential advantages: it could make the compiled Python binary faster for real programs, it could make our benchmark results less noisy since the compiler would be doing a better and most consistent job of generating optimal code.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-130702

先檢視目前的 --enable-optimizations PGO 工作和 ./python -X perf -m test --pgo 命令，包括 Lib/test/support/init.py 中的 test.support.gc_collect，以及涉及 Lib/json/encoder.py 的剖析觀察結果。完成的標準是定義並驗證一個具代表性的替代工作負載，然後將其剖析結果和基準測試影響與現有的單元測試工作進行比較。