Python program running slower inside Gvisor sandbox with ARM64
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
### Description
Hello,
We are currently benchmarking the cpu performance of gvisor compared to normal docker, and found out that same Python program running in gvisor is consistently slower compared to running on native kernel, or even with docker.
Note that we are aware of overhead introduced by additional hook for syscalls, but we are testing the cpu performance, and our test script does not issue syscalls.
The largest difference we observed so far are running on AWS `c6gd.2xlarge` instance. However, when running same suite on c7 instance family, the performance of gvisor is close to native kernel. Thus we are wondering what might be the rootcause of this, and how can we configure gvisor to make it perform better.
Test Environment: AWS `c6gd.2xlarge` instance with AL2 ami. Python. Python version: 3.7.16
Test script (Very simple pi calculation):
```
import time
def calculate_pi(n):
pi = 0
sign = 1
for i in range(1, n * 2, 2):
pi += sign * (4 / i)
sign *= -1
return pi
if __name__ == "__main__":
iterations = 100000000
start = time.time() * 1000
pi_approx = calculate_pi(iterations)
print(time.time() * 1000 - start)
```
Running on native kernel:
```
$ python3 /tmp/pitest.py
16087.6728515625
```
Running with docker container:
```
$ sudo docker run -v /tmp/pitest.py:/tmp/pitest.py amazonlinux:2 yum install -y python3 && python3 /tmp/pitest.py
16197.6875
```
Running with runsc:
```
$ ./bin/runsc --network=none --rootless --platform=systrap run id-10
17876.38525390625
```
In all three cases, the process is consuming nearly 100% of cpu all time. However, when I use `perf` tool to check the stats, it shows that process started by gvisor runs with around 10~15% slower in terms on `instructions per cycle`:
* Native:
```
27,517,274,919 cycles # 2.482 GHz (29.92%)
88,730,249,747 instructions # 3.22 insn per cycle
# 0.01 stalled cycles per insn (29.94%)
11,087.35 msec cpu-clock # 0.923 CPUs utilized
38,231,913,137 cache-references # 3448.309 M/sec (30.03%)
258,817 cache-misses # 0.001 % of all cache refs (20.02%)
34,731,416 branch-misses (20.02%)
1,036,739,378 stalled-cycles-frontend # 3.77% frontend cycles idle (20.02%)
1,103,210,544 stalled-cycles-backend # 4.01% backend cycles idle (20.02%)
5 sched:sched_switch # 0.000 K/sec
11,080,134,270 sched:sched_stat_runtime # 999.367 M/sec
1 page-faults # 0.000 K/sec
281,234 L1-dcache-load-misses (20.02%)
0 cpu-migrations # 0.000 K/sec
11,086.95 msec task-clock # 0.923 CPUs utilized
27,549,011,550 bus-cycles # 2484.770 M/sec (20.00%)
38,076,864,326 mem_access # 3434.324 M/sec (19.91%)
```
* Gvisor:
```
28,392,817,179 cycles # 2.482 GHz (29.98%)
81,657,764,151 instructions # 2.88 insn per cycle
# 0.04 stalled cycles per insn (30.07%)
11,440.44 msec cpu-clock # 0.953 CPUs utilized
35,172,201,924 cache-references # 3074.430 M/sec (30.15%)
922,867 cache-misses # 0.003 % of all cache refs (20.11%)
32,162,856 branch-misses (20.02%)
670,976,075 stalled-cycles-frontend # 2.36% frontend cycles idle (19.93%)
3,126,416,518 stalled-cycles-backend # 11.01% backend cycles idle (19.93%)
1 sched:sched_switch # 0.000 K/sec
11,440,714,362 sched:sched_stat_runtime # 1000.042 M/sec
1 page-faults # 0.000 K/sec
259,948 L1-dcache-load-misses (19.93%)
0 cpu-migrations # 0.000 K/sec
11,440.04 msec task-clock # 0.952 CPUs utilized
28,426,171,668 bus-cycles # 2484.754 M/sec (19.93%)
35,158,019,280 mem_access # 3073.190 M/sec (19.93%)
```
We suspect that this could be due to memory access delay as on Gvisor case `stalled-cycles-backend` is significantly higher compared to other cases.
### Steps to reproduce
1. Create AWS instance with c6 family. eg. `c6gd.2xlarge`
2. Run the script mentioned above in 3 environments
### runsc version
```shell
runsc version 0.0.0
spec: 1.1.0-rc.1
```
### docker version (if using docker)
_No response_
### uname
5.10.216-204.855.amzn2.aarch64 #1 SMP Sat May 4 16:53:24 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
### kubectl (if using Kubernetes)
_No response_
### repo state (if built from source)
_No response_
### runsc debug logs (if available)
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.