opensearch-scaling-manager

Proposal description

The opensearch-scaling-manager is tool offering auto-scaling facilities to the opensearch clusters. It will be done in the following way:
Horizontally scaling up/down the cluster by adding or removing the nodes from the opensearch clusters. This will happen when the traffic increases and decrease. The opensearch-scaling-manager will analyze the data ingestion and scale up/down automatically.

The initial scope of the project is to develop a solution with scaling up/down by one node with core team and then extend the community support to develop the other feature of it.
The plan is to start developing a solution for AWS platform and slowly extend it to other platforms as well.

Key pain points or opportunity details

Existing customers were facing a difficulty to gracefully scale up/down the opensearch cluster with respect to increase/decrease of the data ingestion and search queries. Hence it is a great opportunity for us to develop a solution which can help customers to solve their problems.

Moreover, we are making it as an open-source project, so core part of the development is done by core team and then we can extend this to open-source community.

Objective of the proposed project

Following are the deliverables for the proposed project:

• Solution to horizontal scale up/down opensearch nodes in the cluster with a given configuration. To start with, we will support AWS platform.
• The solution will be extended to support multiple node scaling and Rule based cluster sizing in the future releases and the support for multiple platforms.

Existing alternatives & why I think they are not enough (with links to those projects)

• https://github.com/zalando-incubator/es-operator - This solution supports operator for running Elasticsearch in Kubernetes. The elastic search cluster will be managed by the es-operator which is not desirable.
• https://github.com/upmc-enterprises/elasticsearch-operator - Though the solution supports operator for running Elasticsearch on AWS, the elastic search cluster will be managed by the es-operator which is not desirable.

Ballpark effort estimate (X engineers for Y Months)

4 Engineers for 3-4 months.

Expected number of users/organizations of the proposed project (Submit supporting material, if it exists).

No files, tests, or entry points are named. Begin by defining the AWS implementation boundary and the configuration and ingestion or query signals described in the proposal; confirm the one-node horizontal scale-up and scale-down behavior with graceful cluster handling. Done is an initial AWS solution meeting the stated single-node scaling objective.