elastic / elastic/rally

Use Elasticsearch's logging configuration

Open
#1,371 6 comments 0 reactions 0 assignees View on GitHub
:Benchmark Candidate Management enhancement
Dominant language
Python
Stars
2k
Forks
348
Avg merge
6d 17h
Merged PRs (30d)
5

Description

Historically Rally needed to modify Elasticsearch's logging configuration, e.g. to enable verbose logging for the `IndexingMemoryController`. We've removed the surrounding infrastructure in Rally a while ago. Therefore, Rally can just use the logging configuration that is shipped with Elasticsearch.

**Implementation notes**

Before applying the configuration in rally-teams, Rally wipes the Elaticsearch's configuration directory:

https://github.com/elastic/rally/blob/24dee9ed704e5c98fc2999a665853ba4b2941586/esrally/mechanic/provisioner.py#L185

Then, it applies the configuration; it writes files in append mode (this allows mixins to contribute to the configuration):

https://github.com/elastic/rally/blob/24dee9ed704e5c98fc2999a665853ba4b2941586/esrally/mechanic/provisioner.py#L156-L166

A possible solution is to delete all but the `log4j2.properties` file and then apply the config as is. We should not hardcode that name though but instead introduce a "allowlist" concept (files to keep from the original config). The allowlist would be defined in rally-teams and evaluated by Rally. For backwards-compatibility we should also explicitly skip copying any files on the allowlist.

We also have a dedicated provisioner for the Docker image which writes config files to a directory on the host and then provides each file via a mount:

https://github.com/elastic/rally/blob/24dee9ed704e5c98fc2999a665853ba4b2941586/esrally/mechanic/provisioner.py#L466-L476

Here, it is sufficient to only skip copying any file on the allowlist.

Contributor guide

Open the contributing guide

Research direction

Read the configuration handling in esrally/mechanic/provisioner.py at the referenced lines, including the regular and Docker provisioners, and inspect how rally-teams defines Elasticsearch configuration. Trace where the configuration directory is wiped and files are copied. Done means an allowlist preserves selected shipped files, skips copying them for backwards compatibility, and works for both provisioners.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.