opensearch-project / opensearch-project/OpenSearch
[RFC] Streaming Index API
@reta is already working on this.
Since Mar 24, 2023.
- Dominant language
- Java
- Stars
- 13.7k
- Forks
- 3k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 108
Description
Is your feature request related to a problem? Please describe.
Current _bulk indexing API places a high configuration burden on users today to avoid RejectedExecutionException due to TOO_MANY_REQUESTS. This forces the user to "experiment" with bulk block sizes, multi-threading, refresh intervals, etc.
Describe the solution you'd like
The _bulk configuration burden and workflow should be relocated from the user and handled by the server. The user experience should switch to an anxiety free API that enables users to send a "stream" of index requests that is load balanced by the server in a Streaming Index mechansim.
This Streaming Index API mechanism should also handle the "durability" responsibility based on a user defined Durability Policy to determine the following:
- What operations to persist in the TransLog (if any)
- What type of remote storage to use (e.g., long term vs short term)
- What documents / segments to replicate
- Where segments should be replicated
- Level of consistency (e.g., how often to ack)
- https://github.com/opensearch-project/OpenSearch/issues/5001
- https://github.com/opensearch-project/OpenSearch/issues/9065
Describe alternatives you've considered
Continue w/ durability as it is today w/ a document replication model.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.