Use Elasticsearch's logging configuration
- Dominant language
- Python
- Stars
- 2k
- Forks
- 348
- Avg merge
- 6d 17h
- Merged PRs (30d)
- 5
Description
Historically Rally needed to modify Elasticsearch's logging configuration, e.g. to enable verbose logging for the `IndexingMemoryController`. We've removed the surrounding infrastructure in Rally a while ago. Therefore, Rally can just use the logging configuration that is shipped with Elasticsearch.
**Implementation notes**
Before applying the configuration in rally-teams, Rally wipes the Elaticsearch's configuration directory:
https://github.com/elastic/rally/blob/24dee9ed704e5c98fc2999a665853ba4b2941586/esrally/mechanic/provisioner.py#L185
Then, it applies the configuration; it writes files in append mode (this allows mixins to contribute to the configuration):
https://github.com/elastic/rally/blob/24dee9ed704e5c98fc2999a665853ba4b2941586/esrally/mechanic/provisioner.py#L156-L166
A possible solution is to delete all but the `log4j2.properties` file and then apply the config as is. We should not hardcode that name though but instead introduce a "allowlist" concept (files to keep from the original config). The allowlist would be defined in rally-teams and evaluated by Rally. For backwards-compatibility we should also explicitly skip copying any files on the allowlist.
We also have a dedicated provisioner for the Docker image which writes config files to a directory on the host and then provides each file via a mount:
https://github.com/elastic/rally/blob/24dee9ed704e5c98fc2999a665853ba4b2941586/esrally/mechanic/provisioner.py#L466-L476
Here, it is sufficient to only skip copying any file on the allowlist.
Contributor guide
Research direction
Read the configuration handling in esrally/mechanic/provisioner.py at the referenced lines, including the regular and Docker provisioners, and inspect how rally-teams defines Elasticsearch configuration. Trace where the configuration directory is wiped and files are copied. Done means an allowlist preserves selected shipped files, skips copying them for backwards compatibility, and works for both provisioners.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100