Propeller slows down clang ~20%
- Dominant language
- C++
- Stars
- 619
- Forks
- 127
- PR merge metrics
- No merged PRs in 30d
Description
I have tried to reproduce [optimization clang with Propeller](https://github.com/google/autofdo/blob/master/docs/OptimizeClangO3WithPropeller.md).
After all modifications described in #179 and #180, modified https://github.com/google/autofdo/blob/master/propeller_optimize_clang.sh started to work on my machine.
Unfortunately, the results looks very strange. Applying propeller to clang slows it down about 20%:
```
BASELINE (samples=3):
Performance counter stats for 'bash -c numactl -C 0-75 ninja -j76 clang && ninja clean' (3 runs):
29910755307348 instructions:u # 1.19 insn per cycle ( +- 0.00% )
25108949747468 cycles:u ( +- 0.01% )
1776069921406 L1-icache-misses:u ( +- 0.00% )
7994150086 iTLB-misses:u ( +- 0.01% )
124.271 +- 0.242 seconds time elapsed ( +- 0.19% )
PROPELLER (samples=3):
Performance counter stats for 'bash -c numactl -C 0-75 ninja -j76 clang && ninja clean' (3 runs):
30835896384964 instructions:u # 1.01 insn per cycle ( +- 0.00% )
30547030491210 cycles:u ( +- 0.01% )
2491256701229 L1-icache-misses:u ( +- 0.00% )
7456817103 iTLB-misses:u ( +- 0.05% )
148.1088 +- 0.0623 seconds time elapsed ( +- 0.04% )
```
I used numactl to pin threads to HW cores. When I disables pins, the results were improved slightly, but, the gap between baseline and propeller continues to be significant:
```
BASELINE (samples=5):
Performance counter stats for 'bash -c ninja clang && ninja clean' (5 runs):
29911672560885 instructions:u # 0.73 insn per cycle ( +- 0.00% )
40762914939742 cycles:u ( +- 0.01% )
2198963872412 L1-icache-misses:u ( +- 0.01% )
16606325255 iTLB-misses:u ( +- 0.05% )
119.413 +- 0.212 seconds time elapsed ( +- 0.18% )
PROPELLER (samples=5):
Performance counter stats for 'bash -c ninja clang && ninja clean' (5 runs):
30835273549813 instructions:u # 0.63 insn per cycle ( +- 0.00% )
49008268336239 cycles:u ( +- 0.01% )
3025079343587 L1-icache-misses:u ( +- 0.02% )
16944457932 iTLB-misses:u ( +- 0.03% )
139.041 +- 0.250 seconds time elapsed ( +- 0.18% )
```
In the case of pinned threads, propeller slightly decreased iTLB misses, while L1-icache-misses increased about 1.5x times.
Tested in RAM-disk on $ lscpu
```
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 152
On-line CPU(s) list: 0-151
Thread(s) per core: 2
Core(s) per socket: 38
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8368 CPU @ 2.40GHz
Stepping: 6
CPU MHz: 3400.000
CPU max MHz: 3400.0000
CPU min MHz: 800.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 48K
L1i cache: 32K
L2 cache: 1280K
L3 cache: 58368K
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126,128,130,132,134,136,138,140,142,144,146,148,150
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111,113,115,117,119,121,123,125,127,129,131,133,135,137,139,141,143,145,147,149,151
```
Used OS:
```
$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.8 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.8 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.8"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.8"
```
Used linux kernel:
```
$ uname -a
Linux XXX.XXX.XXX.XXX 4.18.0-477.21.1.el8_8.x86_64 #1 SMP Tue Aug 8 21:30:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```
Gists:
with numactl:
https://gist.github.com/foxtran/b7fedfbb0bd036629448ce62d18bd7a6
without numactl:
https://gist.github.com/foxtran/fdc4abf8e2de127800f670b9edeeb9f2
Applied patches (with #179, #180):
for numactl:
```diff
--- propeller_optimize_clang.sh.orig 2023-10-26 16:54:07.550679311 +0900
+++ propeller_optimize_clang.sh 2023-10-25 20:10:02.862877767 +0900
@@ -30,25 +30,25 @@ PATH_TO_TRUNK_LLVM_BUILD=${BASE_PROPELLE
PATH_TO_TRUNK_LLVM_INSTALL=${BASE_PROPELLER_CLANG_DIR}/trunk_llvm_install
# Build Trunk LLVM
mkdir -p ${PATH_TO_LLVM_SOURCES} && cd ${PATH_TO_LLVM_SOURCES}
-git clone git@github.com:llvm/llvm-project.git
+git clone -b release/17.x --single-branch https://github.com/llvm/llvm-project.git
mkdir -p ${PATH_TO_TRUNK_LLVM_BUILD} && cd ${PATH_TO_TRUNK_LLVM_BUILD}
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 \
-DCMAKE_INSTALL_PREFIX="${PATH_TO_TRUNK_LLVM_INSTALL}" \
-DLLVM_ENABLE_RTTI=On -DLLVM_INCLUDE_TESTS=Off \
-DLLVM_ENABLE_PROJECTS="clang;lld" ${PATH_TO_LLVM_SOURCES}/llvm-project/llvm
-ninja install
+numactl -C 0-75 ninja -j76 install
#Build create_llvm_prof
PATH_TO_CREATE_LLVM_PROF=${BASE_PROPELLER_CLANG_DIR}/create_llvm_prof_build
mkdir -p ${PATH_TO_CREATE_LLVM_PROF} && cd ${PATH_TO_CREATE_LLVM_PROF}
-git clone --recursive git@github.com:google/autofdo.git
+git clone --recursive https://github.com/google/autofdo.git
mkdir build && cd build
cmake -G Ninja -DCMAKE_INSTALL_PREFIX="." \
-DCMAKE_C_COMPILER="${PATH_TO_TRUNK_LLVM_INSTALL}/bin/clang" \
-DCMAKE_CXX_COMPILER="${PATH_TO_TRUNK_LLVM_INSTALL}/bin/clang++" \
-DLLVM_PATH="${PATH_TO_TRUNK_LLVM_INSTALL}" ../autofdo/
-ninja
+numactl -C 0-75 ninja -j76
ls create_llvm_prof
# Common CMAKE Flags
@@ -70,7 +70,7 @@ BASELINE_CC_LD_CMAKE_FLAGS=(
PATH_TO_BASELINE_CLANG_BUILD=${BASE_PROPELLER_CLANG_DIR}/baseline_clang_build
mkdir -p ${PATH_TO_BASELINE_CLANG_BUILD} && cd ${PATH_TO_BASELINE_CLANG_BUILD}
cmake -G Ninja "${COMMON_CMAKE_FLAGS[@]}" "${BASELINE_CC_LD_CMAKE_FLAGS[@]}" ${PATH_TO_LLVM_SOURCES}/llvm-project/llvm
-ninja clang
+numactl -C 0-75 ninja -j76 clang
# Labels CMAKE Flags
LABELS_CC_LD_CMAKE_FLAGS=(
@@ -84,13 +84,13 @@ LABELS_CC_LD_CMAKE_FLAGS=(
PATH_TO_LABELS_CLANG_BUILD=${BASE_PROPELLER_CLANG_DIR}/labels_clang_build
mkdir -p ${PATH_TO_LABELS_CLANG_BUILD} && cd ${PATH_TO_LABELS_CLANG_BUILD}
cmake -G Ninja "${COMMON_CMAKE_FLAGS[@]}" "${LABELS_CC_LD_CMAKE_FLAGS[@]}" ${PATH_TO_LLVM_SOURCES}/llvm-project/llvm
-ninja clang
+numactl -C 0-75 ninja -j76 clang
# Set up Benchmarking and BUILD
BENCHMARKING_CLANG_BUILD=${BASE_PROPELLER_CLANG_DIR}/benchmarking_clang_build
mkdir -p ${BENCHMARKING_CLANG_BUILD} && cd ${BENCHMARKING_CLANG_BUILD}
mkdir -p symlink_to_clang_binary && cd symlink_to_clang_binary
-CLANG_VERSION=$(sed -Ene 's!^CLANG_EXECUTABLE_VERSION:STRING=(.*)$!\1!p' ${PATH_TO_TRUNK_LLVM_BUILD}/CMakeCache.txt)
+CLANG_VERSION=$(sed -Ene 's!^CLANG_EXECUTABLE_VERSION:STRING=(.*)$!\1!p' ${PATH_TO_TRUNK_LLVM_BUILD}/CMakeCache.txt) #'
ln -sf ${PATH_TO_LABELS_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang
ln -sf ${PATH_TO_LABELS_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang++
@@ -111,6 +111,7 @@ ls perf.data
cd ${BENCHMARKING_CLANG_BUILD}
${PATH_TO_CREATE_LLVM_PROF}/build/create_llvm_prof --format=propeller \
--binary=${PATH_TO_LABELS_CLANG_BUILD}/bin/clang-${CLANG_VERSION} \
+ --profiled_binary_name=${PATH_TO_LABELS_CLANG_BUILD}/bin/clang-${CLANG_VERSION} \
--profile=perf.data --out=cluster.txt --propeller_symorder=symorder.txt 2>/dev/null 1>/dev/null
ls cluster.txt symorder.txt
@@ -126,7 +127,7 @@ PROPELLER_CC_LD_CMAKE_FLAGS=(
PATH_TO_PROPELLER_CLANG_BUILD=${BASE_PROPELLER_CLANG_DIR}/propeller_build
mkdir -p ${PATH_TO_PROPELLER_CLANG_BUILD} && cd ${PATH_TO_PROPELLER_CLANG_BUILD}
cmake -G Ninja "${COMMON_CMAKE_FLAGS[@]}" "${PROPELLER_CC_LD_CMAKE_FLAGS[@]}" ${PATH_TO_LLVM_SOURCES}/llvm-project/llvm
-ninja clang
+numactl -C 0-75 ninja -j76 clang
# Run comparison of baseline verus propeller optimized clang
cd ${BENCHMARKING_CLANG_BUILD}/symlink_to_clang_binary
@@ -134,11 +135,11 @@ ln -sf ${PATH_TO_BASELINE_CLANG_BUILD}/b
ln -sf ${PATH_TO_BASELINE_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang++
cd ..
ninja clean
-perf stat -r5 -e instructions,cycles,L1-icache-misses,iTLB-misses -- bash -c "ninja -j48 clang && ninja clean"
+perf stat -r5 -e instructions,cycles,L1-icache-misses,iTLB-misses -- bash -c "numactl -C 0-75 ninja -j76 clang && ninja clean"
cd ${BENCHMARKING_CLANG_BUILD}/symlink_to_clang_binary
ln -sf ${PATH_TO_PROPELLER_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang
ln -sf ${PATH_TO_PROPELLER_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang++
cd ..
ninja clean
-perf stat -r5 -e instructions,cycles,L1-icache-misses,iTLB-misses -- bash -c "ninja -j48 clang && ninja clean"
+perf stat -r5 -e instructions,cycles,L1-icache-misses,iTLB-misses -- bash -c "numactl -C 0-75 ninja -j76 clang && ninja clean"
```
Without numactl:
```diff
--- propeller_optimize_clang.sh.orig 2023-10-26 16:54:07.550679311 +0900
+++ propeller_optimize_clang-nopins.sh 2023-10-25 20:23:21.399123785 +0900
@@ -30,7 +30,7 @@ PATH_TO_TRUNK_LLVM_BUILD=${BASE_PROPELLE
PATH_TO_TRUNK_LLVM_INSTALL=${BASE_PROPELLER_CLANG_DIR}/trunk_llvm_install
# Build Trunk LLVM
mkdir -p ${PATH_TO_LLVM_SOURCES} && cd ${PATH_TO_LLVM_SOURCES}
-git clone git@github.com:llvm/llvm-project.git
+git clone -b release/17.x --single-branch https://github.com/llvm/llvm-project.git
mkdir -p ${PATH_TO_TRUNK_LLVM_BUILD} && cd ${PATH_TO_TRUNK_LLVM_BUILD}
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 \
-DCMAKE_INSTALL_PREFIX="${PATH_TO_TRUNK_LLVM_INSTALL}" \
@@ -42,7 +42,7 @@ ninja install
PATH_TO_CREATE_LLVM_PROF=${BASE_PROPELLER_CLANG_DIR}/create_llvm_prof_build
mkdir -p ${PATH_TO_CREATE_LLVM_PROF} && cd ${PATH_TO_CREATE_LLVM_PROF}
-git clone --recursive git@github.com:google/autofdo.git
+git clone --recursive https://github.com/google/autofdo.git
mkdir build && cd build
cmake -G Ninja -DCMAKE_INSTALL_PREFIX="." \
-DCMAKE_C_COMPILER="${PATH_TO_TRUNK_LLVM_INSTALL}/bin/clang" \
@@ -90,7 +90,7 @@ ninja clang
BENCHMARKING_CLANG_BUILD=${BASE_PROPELLER_CLANG_DIR}/benchmarking_clang_build
mkdir -p ${BENCHMARKING_CLANG_BUILD} && cd ${BENCHMARKING_CLANG_BUILD}
mkdir -p symlink_to_clang_binary && cd symlink_to_clang_binary
-CLANG_VERSION=$(sed -Ene 's!^CLANG_EXECUTABLE_VERSION:STRING=(.*)$!\1!p' ${PATH_TO_TRUNK_LLVM_BUILD}/CMakeCache.txt)
+CLANG_VERSION=$(sed -Ene 's!^CLANG_EXECUTABLE_VERSION:STRING=(.*)$!\1!p' ${PATH_TO_TRUNK_LLVM_BUILD}/CMakeCache.txt) #'
ln -sf ${PATH_TO_LABELS_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang
ln -sf ${PATH_TO_LABELS_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang++
@@ -111,6 +111,7 @@ ls perf.data
cd ${BENCHMARKING_CLANG_BUILD}
${PATH_TO_CREATE_LLVM_PROF}/build/create_llvm_prof --format=propeller \
--binary=${PATH_TO_LABELS_CLANG_BUILD}/bin/clang-${CLANG_VERSION} \
+ --profiled_binary_name=${PATH_TO_LABELS_CLANG_BUILD}/bin/clang-${CLANG_VERSION} \
--profile=perf.data --out=cluster.txt --propeller_symorder=symorder.txt 2>/dev/null 1>/dev/null
ls cluster.txt symorder.txt
@@ -134,11 +135,11 @@ ln -sf ${PATH_TO_BASELINE_CLANG_BUILD}/b
ln -sf ${PATH_TO_BASELINE_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang++
cd ..
ninja clean
-perf stat -r5 -e instructions,cycles,L1-icache-misses,iTLB-misses -- bash -c "ninja -j48 clang && ninja clean"
+perf stat -r5 -e instructions,cycles,L1-icache-misses,iTLB-misses -- bash -c "ninja clang && ninja clean"
cd ${BENCHMARKING_CLANG_BUILD}/symlink_to_clang_binary
ln -sf ${PATH_TO_PROPELLER_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang
ln -sf ${PATH_TO_PROPELLER_CLANG_BUILD}/bin/clang-${CLANG_VERSION} clang++
cd ..
ninja clean
-perf stat -r5 -e instructions,cycles,L1-icache-misses,iTLB-misses -- bash -c "ninja -j48 clang && ninja clean"
+perf stat -r5 -e instructions,cycles,L1-icache-misses,iTLB-misses -- bash -c "ninja clang && ninja clean"
```
Contributor guide
Assessment
This issue has not been assessed yet.