opensearch-project / opensearch-project/OpenSearch

[RFC] Streaming Index API

Open
#3,000 23 comments 7 reactions 1 assignee View on GitHub

@reta is already working on this.

Since Mar 24, 2023.

discuss feature Indexing lucene RFC Roadmap:Cost/Performance/Scale
Dominant language
Java
Stars
13.7k
Forks
3k
Avg merge
2d 23h
Merged PRs (30d)
108

Description

Is your feature request related to a problem? Please describe.
Current _bulk indexing API places a high configuration burden on users today to avoid RejectedExecutionException due to TOO_MANY_REQUESTS. This forces the user to "experiment" with bulk block sizes, multi-threading, refresh intervals, etc.

Describe the solution you'd like
The _bulk configuration burden and workflow should be relocated from the user and handled by the server. The user experience should switch to an anxiety free API that enables users to send a "stream" of index requests that is load balanced by the server in a Streaming Index mechansim.

This Streaming Index API mechanism should also handle the "durability" responsibility based on a user defined Durability Policy to determine the following:

  1. What operations to persist in the TransLog (if any)
  2. What type of remote storage to use (e.g., long term vs short term)
  3. What documents / segments to replicate
  4. Where segments should be replicated
  5. Level of consistency (e.g., how often to ack)

Describe alternatives you've considered
Continue w/ durability as it is today w/ a document replication model.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.