elastic / elastic/kibana-load-testing

gather Elasticsearch and Kibana monitoring data during Cloud load test runs

Open
#38 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
6
Forks
18
PR merge metrics
No merged PRs in 30d

Description

Similar to https://github.com/elastic/kibana-load-testing/issues/39
It would be really useful to get the monitoring metrics during load test runs. This would show things like the Elasticsearch search queue and rejected searches. It would also show Kibana memory and response times.

It would be a little challenging to look at the Monitoring page along side of our dashboard so we might re-build some of the monitoring visualizations from the monitoring indices so we can add them to one dashboard.

The problem is, we want to run load tests on the latest master snapshot builds and those are only deployed to Cloud staging. But our kibana-stats.elastic.dev is on Cloud production. We can't configure our load test deployment to send monitoring data from staging to a deployment id in production.
We would need to configure our load test deployment to use the kibana-stats URL, username, password and we don't think we have a way to do that right now.
https://www.elastic.co/guide/en/cloud/current/Deployment_-_CRUD.html#ec_request_example_10

We thought about configuring the load test deployment to keep the monitoring data local and at the end of the job, save a snapshot to cloud storage, and immediately restore that snapshot into kibana-stats.

We also thought about running kibana-stats in Cloud staging so that we could use the deployment id.

/cc @marius-dr @dmlemeshko

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the related kibana-load-testing issue 39 and the Cloud CRUD API monitoring configuration example linked in this issue. Compare the staging and production deployment constraints and the proposed local snapshot approach. Done means load-test runs expose Elasticsearch and Kibana monitoring metrics in a usable dashboard.

Written by the indexing model from the issue text.

Assessment

Tech stack
cloud, elasticsearch, scala
Domain
observability-sre, performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.