4paradigm / 4paradigm/OpenMLDB

Evaluate using Profile-Guided Optimization (PGO) and LLVM BOLT

Đang mở
#3,575 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
build enhancement
Ngôn ngữ chính
C++
Star
1.7k
Fork
331
Merge trung bình
12 ngày 12 giờ
Pull request đã merge (30 ngày)
1

Mô tả

Hi!

Recently I did many Profile-Guided Optimization (PGO) benchmarks on multiple projects (including many databases like PostgreSQL, ClickHouse, Redis, MongoDB, etc.) - the results are available [here](https://github.com/zamazan4ik/awesome-pgo). So that's why I think it's worth trying to apply PGO to OpenMLDB to improve the database performance further.

I can suggest the following things to do:

* Evaluate PGO's results on OpenMLDB.
* If PGO helps to achieve better performance - add a note to OpenMLDB's documentation about that (probably somewhere in the README file). In this case, users and maintainers will be aware of another optimization opportunity for OpenMLDB.
* Provide PGO integration into the build scripts. It can help users and maintainers easily apply PGO for their own workloads.
* Optimize prebuilt binaries with PGO.

Here are some examples of how PGO is already integrated into other projects' build scripts:

* Rustc: a CI [script](https://github.com/rust-lang/rust/blob/master/src/ci/stage-build.py) for the multi-stage build
* GCC:
- Official [docs](https://gcc.gnu.org/install/build.html), section "Building with profile feedback" (even AutoFDO build is supported)
- A [part](https://github.com/gcc-mirror/gcc/blob/4832767db7897be6fb5cbc44f079482c90cb95a6/configure#L7818) in a "wonderful" `configure` script
* Clang: [Docs](https://llvm.org/docs/HowToBuildWithPGO.html)
* Python:
- CPython: [README](https://github.com/python/cpython#profile-guided-optimization)
- Pyston: [README](https://github.com/pyston/pyston#building)
* Go: [Bash script](https://github.com/golang/go/blob/master/src/cmd/compile/profile.sh)
* V8: [Bazel flag](https://github.com/v8/v8/blob/main/BUILD.gn#L184)
* ChakraCore: [Scripts](https://github.com/chakra-core/ChakraCore/tree/master/Build/scripts/pgo)
* Chromium: [Script](https://chromium.googlesource.com/chromium/src/build/config/+/refs/heads/main/compiler/pgo/BUILD.gn)
* Firefox: [Docs](https://firefox-source-docs.mozilla.org/build/buildsystem/pgo.html)
- Thunderbird has PGO support too
* PHP - [Makefile command](https://github.com/php/php-src/blob/master/build/Makefile.global#L138) and old Centminmod [scripts](https://github.com/centminmod/php_pgo_training_scripts)
* MySQL: [CMake script](https://github.com/mysql/mysql-server/blob/8.0/cmake/fprofile.cmake)
* YugabyteDB: [GitHub commit](https://github.com/yugabyte/yugabyte-db/commit/34cb791ed9d3d5f8ae9a9b9e9181a46485e1981d)
* FoundationDB: [Script](https://github.com/apple/foundationdb/blob/1a6114a66f3de508c0cf0a45f72f3687ba05750c/contrib/generate_profile.sh)
* Zstd: [Makefile](https://github.com/facebook/zstd/blob/dev/programs/Makefile#L232)
* [Foot](https://codeberg.org/dnkl/foot): [Scripts](https://codeberg.org/dnkl/foot/src/branch/master/pgo)
* Windows Terminal: [GitHub PR](https://github.com/microsoft/terminal/pull/10071)
* Pydantic-core: [GitHub PR](https://github.com/pydantic/pydantic-core/pull/741)

After PGO, I can suggest evaluating [LLVM BOLT](https://github.com/llvm/llvm-project/blob/main/bolt/README.md) as an additional optimization step after PGO.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Review the project's existing CMake or build scripts to understand the current compilation pipeline. Examine the linked examples of PGO integration in other projects (e.g., MySQL's CMake script, Rust's CI script) to see how profile-guided optimization is typically added. Determine what benchmark or workload would be needed to generate profiling data for OpenMLDB. Success means having a working PGO build process and documented performance improvements.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
android, ansible, apache, arch-linux, aws, azure, babel, blender, bootstrap, c, centos, clickhouse, cmake, cpp, cypress, debian, docker, drupal, elasticsearch, eslint, figma, firebase, gcp, git, github, github-actions, gitlab, go, grafana, graphql, grpc, hadoop, helm, hugo, ios, jekyll, jquery, kafka, kubernetes, linux, macos, matplotlib, mongodb, mysql, neovim, nginx, nodejs, numpy, openapi, pandas, php, playwright, postgresql, postman, prometheus, python, pytorch, rabbitmq, redis, rust, scikit-learn, spark, sqlite, storybook, supabase, tailwindcss, tensorflow, terraform, ubuntu, vite, vscode, webpack, wordpress
Lĩnh vực
ai, backend, backend-api-design, build-system, databases, devtools, machine-learning, observability-sre, performance, testing-qa
Loại issue
Tính năng
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Đình trệ
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
30/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.