apache / apache/gluten

Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT

Open
#3,478 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Recently I checked Profile-Guided Optimization (PGO) improvements on many projects. All current results are available [here](https://github.com/zamazan4ik/awesome-pgo/). E.g. ClickHouse PGO results can be checked [here](https://github.com/ClickHouse/ClickHouse/issues/44567#issuecomment-1589541199). According to the multiple tests, PGO can help with improving performance in many cases. That's why I think trying to optimize the Gluten with PGO can be a good idea.

I can suggest the following action points:
* Perform PGO benchmarks on Gluten. And if it shows improvements - add a note about possible improvements in Gluten performance with PGO.
* Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users and maintainers since they will be able to optimize Gluten according to their own workloads.
* Optimize pre-built binaries

Since the Gluten native part is the library, I think the Pydantic-core [experience](https://github.com/pydantic/pydantic-core/blob/main/Makefile#L66) can be reused here. Also, Clang [supports](https://clang.llvm.org/docs/UsersManual.html#profiling-with-instrumentation) PGO for shared libraries.

Maybe testing Post-Link Optimization techniques (like [LLVM BOLT](https://github.com/llvm/llvm-project/blob/main/bolt/README.md)) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

Here are some examples of how PGO optimization is integrated in other projects:
* Rustc: a CI [script](https://github.com/rust-lang/rust/blob/master/src/ci/stage-build.py) for the multi-stage build
* GCC:
- Official [docs](https://gcc.gnu.org/install/build.html), section "Building with profile feedback" (even AutoFDO build is supported)
- A [part](https://github.com/gcc-mirror/gcc/blob/4832767db7897be6fb5cbc44f079482c90cb95a6/configure#L7818) in a "wonderful" `configure` script
* Clang: [Docs](https://llvm.org/docs/HowToBuildWithPGO.html)
* Python:
- CPython: [README](https://github.com/python/cpython#profile-guided-optimization)
- Pyston: [README](https://github.com/pyston/pyston#building)
* Go: [Bash script](https://github.com/golang/go/blob/master/src/cmd/compile/profile.sh)
* V8: [Bazel flag](https://github.com/v8/v8/blob/main/BUILD.gn#L184)
* ChakraCore: [Scripts](https://github.com/chakra-core/ChakraCore/tree/master/Build/scripts/pgo)
* Chromium: [Script](https://chromium.googlesource.com/chromium/src/build/config/+/refs/heads/main/compiler/pgo/BUILD.gn)
* Firefox: [Docs](https://firefox-source-docs.mozilla.org/build/buildsystem/pgo.html)
- Thunderbird has PGO support too
* PHP - [Makefile command](https://github.com/php/php-src/blob/master/build/Makefile.global#L138) and old Centminmod [scripts](https://github.com/centminmod/php_pgo_training_scripts)
* MySQL: [CMake script](https://github.com/mysql/mysql-server/blob/8.0/cmake/fprofile.cmake)
* YugabyteDB: [GitHub commit](https://github.com/yugabyte/yugabyte-db/commit/34cb791ed9d3d5f8ae9a9b9e9181a46485e1981d)
* FoundationDB: [Script](https://github.com/apple/foundationdb/blob/1a6114a66f3de508c0cf0a45f72f3687ba05750c/contrib/generate_profile.sh)
* Zstd: [Makefile](https://github.com/facebook/zstd/blob/dev/programs/Makefile#L232)
* [Foot](https://codeberg.org/dnkl/foot): [Scripts](https://codeberg.org/dnkl/foot/src/branch/master/pgo)
* Windows Terminal: [GitHub PR](https://github.com/microsoft/terminal/pull/10071)
* Pydantic-core: [GitHub PR](https://github.com/pydantic/pydantic-core/pull/741)
* file.d: [GitHub PR](https://github.com/ozontech/file.d/pull/469)
* OceanBase: [CMake flag](https://github.com/oceanbase/oceanbase/blob/master/cmake/Env.cmake#L55)

Contributor guide

Open the contributing guide

Research direction

No Gluten files, tests, or benchmark entry points are named. Start by locating the native build scripts and performance benchmarks, then define a representative workload for PGO; done would require measured results and a scoped decision on PGO support, with LLVM BOLT explicitly secondary.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.