4paradigm / 4paradigm/OpenMLDB
Evaluate using Profile-Guided Optimization (PGO) and LLVM BOLT
- Ngôn ngữ chính
- C++
- Star
- 1.7k
- Fork
- 331
- Merge trung bình
- 12 ngày 12 giờ
- Pull request đã merge (30 ngày)
- 1
Mô tả
Hi!
Recently I did many Profile-Guided Optimization (PGO) benchmarks on multiple projects (including many databases like PostgreSQL, ClickHouse, Redis, MongoDB, etc.) - the results are available [here](https://github.com/zamazan4ik/awesome-pgo). So that's why I think it's worth trying to apply PGO to OpenMLDB to improve the database performance further.
I can suggest the following things to do:
* Evaluate PGO's results on OpenMLDB.
* If PGO helps to achieve better performance - add a note to OpenMLDB's documentation about that (probably somewhere in the README file). In this case, users and maintainers will be aware of another optimization opportunity for OpenMLDB.
* Provide PGO integration into the build scripts. It can help users and maintainers easily apply PGO for their own workloads.
* Optimize prebuilt binaries with PGO.
Here are some examples of how PGO is already integrated into other projects' build scripts:
* Rustc: a CI [script](https://github.com/rust-lang/rust/blob/master/src/ci/stage-build.py) for the multi-stage build
* GCC:
- Official [docs](https://gcc.gnu.org/install/build.html), section "Building with profile feedback" (even AutoFDO build is supported)
- A [part](https://github.com/gcc-mirror/gcc/blob/4832767db7897be6fb5cbc44f079482c90cb95a6/configure#L7818) in a "wonderful" `configure` script
* Clang: [Docs](https://llvm.org/docs/HowToBuildWithPGO.html)
* Python:
- CPython: [README](https://github.com/python/cpython#profile-guided-optimization)
- Pyston: [README](https://github.com/pyston/pyston#building)
* Go: [Bash script](https://github.com/golang/go/blob/master/src/cmd/compile/profile.sh)
* V8: [Bazel flag](https://github.com/v8/v8/blob/main/BUILD.gn#L184)
* ChakraCore: [Scripts](https://github.com/chakra-core/ChakraCore/tree/master/Build/scripts/pgo)
* Chromium: [Script](https://chromium.googlesource.com/chromium/src/build/config/+/refs/heads/main/compiler/pgo/BUILD.gn)
* Firefox: [Docs](https://firefox-source-docs.mozilla.org/build/buildsystem/pgo.html)
- Thunderbird has PGO support too
* PHP - [Makefile command](https://github.com/php/php-src/blob/master/build/Makefile.global#L138) and old Centminmod [scripts](https://github.com/centminmod/php_pgo_training_scripts)
* MySQL: [CMake script](https://github.com/mysql/mysql-server/blob/8.0/cmake/fprofile.cmake)
* YugabyteDB: [GitHub commit](https://github.com/yugabyte/yugabyte-db/commit/34cb791ed9d3d5f8ae9a9b9e9181a46485e1981d)
* FoundationDB: [Script](https://github.com/apple/foundationdb/blob/1a6114a66f3de508c0cf0a45f72f3687ba05750c/contrib/generate_profile.sh)
* Zstd: [Makefile](https://github.com/facebook/zstd/blob/dev/programs/Makefile#L232)
* [Foot](https://codeberg.org/dnkl/foot): [Scripts](https://codeberg.org/dnkl/foot/src/branch/master/pgo)
* Windows Terminal: [GitHub PR](https://github.com/microsoft/terminal/pull/10071)
* Pydantic-core: [GitHub PR](https://github.com/pydantic/pydantic-core/pull/741)
After PGO, I can suggest evaluating [LLVM BOLT](https://github.com/llvm/llvm-project/blob/main/bolt/README.md) as an additional optimization step after PGO.
Hướng dẫn đóng góp
Hướng nghiên cứu
Review the project's existing CMake or build scripts to understand the current compilation pipeline. Examine the linked examples of PGO integration in other projects (e.g., MySQL's CMake script, Rust's CI script) to see how profile-guided optimization is typically added. Determine what benchmark or workload would be needed to generate profiling data for OpenMLDB. Success means having a working PGO build process and documented performance improvements.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- android, ansible, apache, arch-linux, aws, azure, babel, blender, bootstrap, c, centos, clickhouse, cmake, cpp, cypress, debian, docker, drupal, elasticsearch, eslint, figma, firebase, gcp, git, github, github-actions, gitlab, go, grafana, graphql, grpc, hadoop, helm, hugo, ios, jekyll, jquery, kafka, kubernetes, linux, macos, matplotlib, mongodb, mysql, neovim, nginx, nodejs, numpy, openapi, pandas, php, playwright, postgresql, postman, prometheus, python, pytorch, rabbitmq, redis, rust, scikit-learn, spark, sqlite, storybook, supabase, tailwindcss, tensorflow, terraform, ubuntu, vite, vscode, webpack, wordpress
- Lĩnh vực
- ai, backend, backend-api-design, build-system, databases, devtools, machine-learning, observability-sre, performance, testing-qa
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 30/100