elastic / elastic/apm-agent-nodejs

simplify how our benchmarks are run

Open
#3,384 0 comments 0 reactions 0 assignees View on GitHub
agent-nodejs technical debt
Dominant language
JavaScript
Stars
594
Forks
244
Avg merge
1d 8h
Merged PRs (30d)
16

Description

Once https://github.com/elastic/apm-agent-nodejs/pull/3374 goes in our "run benchmarks in CI" will be roughly this process:

1. apm-agent-nodejs/.github/workflows/microbenchmark.yml
- triggers on commits to main
- this has a "microbenchmark" job with a first step to run
`elastic/apm-pipeline-library/.github/actions/buildkite@current`
with these options:
```
pipeline: apm-agent-microbenchmark
waitFor: true
buildEnvVars: |
script=.ci/scripts/bench.sh
repo=apm-agent-nodejs
sha=${{ github.sha }}
BRANCH_NAME=${{ github.ref_name }}
...
```
Q: is "@current" the same as latest "main" in that repo?
2. apm-pipeline-library/.github/actions/buildkite/action.yml:
- get some secrets from vault (builtkite API access token)
- calls its local "./run.sh"
3. apm-pipeline-library/.github/actions/buildkite/run.sh:
- labouriously put together a POST body
- `curl https://api.buildkite.com/v2/organizations/elastic/pipelines/apm-agent-microbenchmark/builds -X POST -d "$JSON"`
to start a build.
Q: Where is that `apm-agent-microbenchmark` pipeline defined?
- spin loop (1s) on Buildkite build URL waiting for build to finish.
- download and dump all the build logs to stdout
4. (apm-agent-microbenchmark pipeline)
- I'm not sure where this is defined, but presumably it is configured
to run "observability-internal/.buildkite/scripts/microbenchmark.sh"
5. "observability-internal/.buildkite/scripts/microbenchmark.sh" (private)
- checks out the $repo and $sha
- executes the $script
6. "apm-agent-nodejs/.ci/scripts/bench.sh"
- minimal env setup for the Builtkite runner env
- `.ci/scripts/run-benchmarks.sh "apm-agent-benchmark-results.json" "14"`
- uses `curl` with some magic envvars defined in "observability-robots/.buildkite/hooks/pre-command"
(pulled from vault) to upload benchmark data to Elasticsearch
7. "apm-agent-nodejs/.ci/scripts/run-benchmarks.sh"
```
source ./${SCRIPTPATH}/prepare-benchmarks-env.sh
npm run bench:ci "${RESULT_FILE}" "${NODE_VERSION}"
```
8. "apm-agent-nodejs/.ci/scripts/prepare-benchmarks-env.sh"
- install nvm
- install the node version given in the command above (see `"14"`)
9. `npm run bench:ci` -> "apm-agent-nodejs/test/benchmarks/scripts/run-benchmarks-ci.sh"
- sets up CPU env for benchmarking (`cpufreq-set`, `cset`)
- calls "apm-agent-nodejs/test/benchmarks/run-benchmarks.sh"
(Note that this is the *second* script called "run-benchmarks.sh")
10. "apm-agent-nodejs/test/benchmarks/run-benchmarks.sh"
- possibly call `.ci/scripts/prepare-benchmarks-env.sh` **again**, `if [ -n "${SUDO_COMMAND}" ]`
- then run each `test/benchmarks/NNN-*.js` benchmark script with a mock APM server

This various scripts in this repo for this could be spectacularly simpler. It is very confusing to follow currently. This issue is about improving this... and perhaps documenting it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.