elastic / elastic/apm-agent-nodejs
simplify how our benchmarks are run
- Dominant language
- JavaScript
- Stars
- 594
- Forks
- 244
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 16
Description
Once https://github.com/elastic/apm-agent-nodejs/pull/3374 goes in our "run benchmarks in CI" will be roughly this process:
1. apm-agent-nodejs/.github/workflows/microbenchmark.yml
- triggers on commits to main
- this has a "microbenchmark" job with a first step to run
`elastic/apm-pipeline-library/.github/actions/buildkite@current`
with these options:
```
pipeline: apm-agent-microbenchmark
waitFor: true
buildEnvVars: |
script=.ci/scripts/bench.sh
repo=apm-agent-nodejs
sha=${{ github.sha }}
BRANCH_NAME=${{ github.ref_name }}
...
```
Q: is "@current" the same as latest "main" in that repo?
2. apm-pipeline-library/.github/actions/buildkite/action.yml:
- get some secrets from vault (builtkite API access token)
- calls its local "./run.sh"
3. apm-pipeline-library/.github/actions/buildkite/run.sh:
- labouriously put together a POST body
- `curl https://api.buildkite.com/v2/organizations/elastic/pipelines/apm-agent-microbenchmark/builds -X POST -d "$JSON"`
to start a build.
Q: Where is that `apm-agent-microbenchmark` pipeline defined?
- spin loop (1s) on Buildkite build URL waiting for build to finish.
- download and dump all the build logs to stdout
4. (apm-agent-microbenchmark pipeline)
- I'm not sure where this is defined, but presumably it is configured
to run "observability-internal/.buildkite/scripts/microbenchmark.sh"
5. "observability-internal/.buildkite/scripts/microbenchmark.sh" (private)
- checks out the $repo and $sha
- executes the $script
6. "apm-agent-nodejs/.ci/scripts/bench.sh"
- minimal env setup for the Builtkite runner env
- `.ci/scripts/run-benchmarks.sh "apm-agent-benchmark-results.json" "14"`
- uses `curl` with some magic envvars defined in "observability-robots/.buildkite/hooks/pre-command"
(pulled from vault) to upload benchmark data to Elasticsearch
7. "apm-agent-nodejs/.ci/scripts/run-benchmarks.sh"
```
source ./${SCRIPTPATH}/prepare-benchmarks-env.sh
npm run bench:ci "${RESULT_FILE}" "${NODE_VERSION}"
```
8. "apm-agent-nodejs/.ci/scripts/prepare-benchmarks-env.sh"
- install nvm
- install the node version given in the command above (see `"14"`)
9. `npm run bench:ci` -> "apm-agent-nodejs/test/benchmarks/scripts/run-benchmarks-ci.sh"
- sets up CPU env for benchmarking (`cpufreq-set`, `cset`)
- calls "apm-agent-nodejs/test/benchmarks/run-benchmarks.sh"
(Note that this is the *second* script called "run-benchmarks.sh")
10. "apm-agent-nodejs/test/benchmarks/run-benchmarks.sh"
- possibly call `.ci/scripts/prepare-benchmarks-env.sh` **again**, `if [ -n "${SUDO_COMMAND}" ]`
- then run each `test/benchmarks/NNN-*.js` benchmark script with a mock APM server
This various scripts in this repo for this could be spectacularly simpler. It is very confusing to follow currently. This issue is about improving this... and perhaps documenting it.
Contributor guide
Assessment
This issue has not been assessed yet.