elastic / elastic/elastic-package
Low disk watermark exceeded in CI pipeline, resulting in NoShardAvailableActionException and failing tests
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
elastic-package version: v0.48.0
Stack versions: 8.2.0-SNAPSHOT, 8.3.0-SNAPSHOT
Pipeline affected: https://apm-ci.elastic.co/job/apm-agent-rum/job/e2e-synthetics-mbp/
Tests that run on this pipeline: https://github.com/elastic/synthetics/blob/main/__tests__/e2e/synthetics.journey.ts
Documentation for these tests: https://github.com/elastic/synthetics/tree/main/__tests__/e2e
Script where elastic-package is invoked: https://github.com/elastic/synthetics/blob/main/__tests__/e2e/scripts/setup_integration.sh
The cluster spun up for Elastic Synthetics e2e tests is reporting low disk watermark exceeded, even when no Synthetics data is indexed.
The cluster is brought up with `elastic-package stack up --version {version}`.
This error was uncovered by extracting the ES logs, both with the Synthetics tests enabled and disabled. When the tests are disabled, ES still reports low disk watermark exceeded:
```
{"@timestamp":"2022-05-07T00:02:29.686Z", "log.level": "INFO", "message":"low disk watermark [85%] exceeded on [EpUAMaKqQaOdKT-scbq1Kg][2e2f4f385b0a][/usr/share/elasticsearch/data] free: 15.2gb[10.4%], replicas will not be assigned to this node", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[2e2f4f385b0a][generic][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor","elasticsearch.node.name":"2e2f4f385b0a","elasticsearch.cluster.name":"elasticsearch"}
```
Full logs: https://apm-ci.elastic.co/job/apm-agent-rum/job/e2e-synthetics-mbp/view/change-requests/job/PR-499/25/consoleFull (Search for `Fetching ES logs`)
When tests are enabled, shards are not allocated for the Synthetics data, resulting in `NoShardAvailableActionException`.
```
{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":null,"index_uuid":"U2HSiqeRQ7SFlqVGPhV73A","shard":"0","index":".ds-synthetics-browser-default-2022.05.06-000001"},{"type":"no_shard_available_action_exception","reason":null,"index_uuid":"8yU95eCjShqTwHQZO3Kyxg","shard":"0","index":".ds-synthetics-http-default-2022.05.06-000001"},{"type":"no_shard_available_action_exception","reason":null,"index_uuid":"bqS1e4RWS3WiJtzFKk3TgQ","shard":"0","index":".ds-synthetics-tcp-default-2022.05.06-000001"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ds-synthetics-browser-default-2022.05.06-000001","node":null,"reason":{"type":"no_shard_available_action_exception","reason":null,"index_uuid":"U2HSiqeRQ7SFlqVGPhV73A","shard":"0","index":".ds-synthetics-browser-default-2022.05.06-000001"}},{"shard":0,"index":".ds-synthetics-http-default-2022.05.06-000001","node":null,"reason":{"type":"no_shard_available_action_exception","reason":null,"index_uuid":"8yU95eCjShqTwHQZO3Kyxg","shard":"0","index":".ds-synthetics-http-default-2022.05.06-000001"}},{"shard":0,"index":".ds-synthetics-tcp-default-2022.05.06-000001","node":null,"reason":{"type":"no_shard_available_action_exception","reason":null,"index_uuid":"bqS1e4RWS3WiJtzFKk3TgQ","shard":"0","index":".ds-synthetics-tcp-default-2022.05.06-000001"}}]},"status":503}
```
Full logs `https://apm-ci.elastic.co/job/apm-agent-rum/job/e2e-synthetics-mbp/view/change-requests/job/PR-499/24/consoleFull` (Search no_shared_available_action_exception or NoShardAvailableActionException)
This has caused Elastic Synthetics e2e tests to fail consistently for the last few days.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with __tests__/e2e/scripts/setup_integration.sh and the elastic-package stack up --version command, then inspect the linked Elasticsearch logs with tests enabled and disabled. Use __tests__/e2e/synthetics.journey.ts to reproduce the failure. Done means the CI cluster no longer reports the low disk watermark or NoShardAvailableActionException and the e2e tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- ci-cd, cli, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100