elastic / elastic/elasticsearch

Unused repository plugins consume a lot of metaspace

Open
#97,482 2 comments 0 reactions 0 assignees View on GitHub
:Core/Infra/Plugins >bug Team:Core/Infra
Dominant language
Java
Stars
77.9k
Forks
26.1k
PR merge metrics
PR metrics pending

Description

From looking into ES's metaspace usage the other day ...

Currently, just firing up a default Elasticsearch node that has no repositories configured, one can see that both the Azure and the S3 repositories use up a lot of metaspace even when unsued.
It's ~4.6M for the S3 plugin and ~6.3M for the Azure plugin. This might be a relevant amount of memory for small deployments, e.g. for a 512M heap deployment, an extra wasted 10M of off-heap memory isn't great.

In both cases the main contributor to these numbers is the Jackson dependency, for the Azure dependency redundant loading the Netty which we have on the classpath from the transport already (but in a separate loader) also adds 2M or so.

I wonder if we could introduce some sort of lazy loading here that would allow us to save those 11MB+ of off-heap memory here. Alternatively, maybe we could share the Netty (same version for Azure and the transport) and Jackson (us and S3 share the version here) classes with those already loaded in the transport/core?

relates https://github.com/elastic/elasticsearch/issues/81652

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the default Elasticsearch node's metaspace usage with no repositories configured, then inspect the S3 and Azure repository plugins and their class-loading dependencies. Done means unused repository plugins no longer consume the reported excess metaspace, with measurements showing the reduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, azure, java
Domain
backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.