4paradigm / 4paradigm/OpenMLDB

Evaluate using Profile-Guided Optimization (PGO) and LLVM BOLT

Offen
#3,575 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
build enhancement
Vorherrschende Sprache
C++
Sterne
1.7k
Forks
331
Ø Merge
12 T. 12 Std.
Gemergte PRs (30 T.)
1

Beschreibung

Hi!

Recently I did many Profile-Guided Optimization (PGO) benchmarks on multiple projects (including many databases like PostgreSQL, ClickHouse, Redis, MongoDB, etc.) - the results are available [here](https://github.com/zamazan4ik/awesome-pgo). So that's why I think it's worth trying to apply PGO to OpenMLDB to improve the database performance further.

I can suggest the following things to do:

* Evaluate PGO's results on OpenMLDB.
* If PGO helps to achieve better performance - add a note to OpenMLDB's documentation about that (probably somewhere in the README file). In this case, users and maintainers will be aware of another optimization opportunity for OpenMLDB.
* Provide PGO integration into the build scripts. It can help users and maintainers easily apply PGO for their own workloads.
* Optimize prebuilt binaries with PGO.

Here are some examples of how PGO is already integrated into other projects' build scripts:

* Rustc: a CI [script](https://github.com/rust-lang/rust/blob/master/src/ci/stage-build.py) for the multi-stage build
* GCC:
- Official [docs](https://gcc.gnu.org/install/build.html), section "Building with profile feedback" (even AutoFDO build is supported)
- A [part](https://github.com/gcc-mirror/gcc/blob/4832767db7897be6fb5cbc44f079482c90cb95a6/configure#L7818) in a "wonderful" `configure` script
* Clang: [Docs](https://llvm.org/docs/HowToBuildWithPGO.html)
* Python:
- CPython: [README](https://github.com/python/cpython#profile-guided-optimization)
- Pyston: [README](https://github.com/pyston/pyston#building)
* Go: [Bash script](https://github.com/golang/go/blob/master/src/cmd/compile/profile.sh)
* V8: [Bazel flag](https://github.com/v8/v8/blob/main/BUILD.gn#L184)
* ChakraCore: [Scripts](https://github.com/chakra-core/ChakraCore/tree/master/Build/scripts/pgo)
* Chromium: [Script](https://chromium.googlesource.com/chromium/src/build/config/+/refs/heads/main/compiler/pgo/BUILD.gn)
* Firefox: [Docs](https://firefox-source-docs.mozilla.org/build/buildsystem/pgo.html)
- Thunderbird has PGO support too
* PHP - [Makefile command](https://github.com/php/php-src/blob/master/build/Makefile.global#L138) and old Centminmod [scripts](https://github.com/centminmod/php_pgo_training_scripts)
* MySQL: [CMake script](https://github.com/mysql/mysql-server/blob/8.0/cmake/fprofile.cmake)
* YugabyteDB: [GitHub commit](https://github.com/yugabyte/yugabyte-db/commit/34cb791ed9d3d5f8ae9a9b9e9181a46485e1981d)
* FoundationDB: [Script](https://github.com/apple/foundationdb/blob/1a6114a66f3de508c0cf0a45f72f3687ba05750c/contrib/generate_profile.sh)
* Zstd: [Makefile](https://github.com/facebook/zstd/blob/dev/programs/Makefile#L232)
* [Foot](https://codeberg.org/dnkl/foot): [Scripts](https://codeberg.org/dnkl/foot/src/branch/master/pgo)
* Windows Terminal: [GitHub PR](https://github.com/microsoft/terminal/pull/10071)
* Pydantic-core: [GitHub PR](https://github.com/pydantic/pydantic-core/pull/741)

After PGO, I can suggest evaluating [LLVM BOLT](https://github.com/llvm/llvm-project/blob/main/bolt/README.md) as an additional optimization step after PGO.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Review the project's existing CMake or build scripts to understand the current compilation pipeline. Examine the linked examples of PGO integration in other projects (e.g., MySQL's CMake script, Rust's CI script) to see how profile-guided optimization is typically added. Determine what benchmark or workload would be needed to generate profiling data for OpenMLDB. Success means having a working PGO build process and documented performance improvements.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
android, ansible, apache, arch-linux, aws, azure, babel, blender, bootstrap, c, centos, clickhouse, cmake, cpp, cypress, debian, docker, drupal, elasticsearch, eslint, figma, firebase, gcp, git, github, github-actions, gitlab, go, grafana, graphql, grpc, hadoop, helm, hugo, ios, jekyll, jquery, kafka, kubernetes, linux, macos, matplotlib, mongodb, mysql, neovim, nginx, nodejs, numpy, openapi, pandas, php, playwright, postgresql, postman, prometheus, python, pytorch, rabbitmq, redis, rust, scikit-learn, spark, sqlite, storybook, supabase, tailwindcss, tensorflow, terraform, ubuntu, vite, vscode, webpack, wordpress
Bereich
ai, backend, backend-api-design, build-system, databases, devtools, machine-learning, observability-sre, performance, testing-qa
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
30/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.