google / google/go-jsonnet

Performance regression between v0.20.0 and master

Open
#784 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.8k
Forks
263
PR merge metrics
No merged PRs in 30d

Description

I ran into this while debugging an other mystery. Master at the time of writing is 923f51b. I tested this on Apple M1 hardware.

It appears that one of the recent changes has had a considerable performance impact, it is reproducible with the cpp benchmarks.

```
$ for i in v0.20.0 master; do git checkout $i; go build -o jsonnet-$i ./cmd/jsonnet; done

$ hyperfine --warmup 3 --parameter-list binary './jsonnet-v0.20.0,./jsonnet-master' '{binary} cpp-jsonnet/benchmarks/bench.07.jsonnet --max-stack 1000000'

Benchmark #1: ./jsonnet-v0.20.0 cpp-jsonnet/benchmarks/bench.07.jsonnet --max-stack 1000000
Time (mean ± σ): 201.7 ms ± 3.8 ms [User: 169.0 ms, System: 46.8 ms]
Range (min … max): 196.3 ms … 208.5 ms 13 runs

Benchmark #2: ./jsonnet-master cpp-jsonnet/benchmarks/bench.07.jsonnet --max-stack 1000000
Time (mean ± σ): 290.9 ms ± 2.3 ms [User: 239.1 ms, System: 71.4 ms]
Range (min … max): 287.6 ms … 294.1 ms 10 runs

Summary
'./jsonnet-v0.20.0 cpp-jsonnet/benchmarks/bench.07.jsonnet --max-stack 1000000' ran
1.44 ± 0.03 times faster than './jsonnet-master cpp-jsonnet/benchmarks/bench.07.jsonnet --max-stack 1000000'
```

Profiling:

```
$ for i in v0.20.0 master; do JSONNET_CPU_PROFILE=cpu-$i.pprof ./jsonnet-$i cpp-jsonnet/benchmarks/bench.07.jsonnet --max-stack 1000000; go tool pprof -svg -output cpu-$i.pprof.svg cpu-$i.pprof; done
```

These are short-lived runs, so I'm not sure how accurate the profiles are.

- [cpu-v0.20.0.pprof.svg](https://github.com/user-attachments/assets/7cd94f39-186f-438a-8e08-12b555867aff)
- [cpu-master.pprof.svg](https://github.com/user-attachments/assets/8c04ad99-0a13-43c8-85d7-ad0b39b9d711)

I also noticed during a larger build that jsonnet-master was using 3.5 cores, whereas jsonnet-v0.20.0 only used a single core.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.