elsa-workflows / elsa-workflows/elsa-core

Use Alias and ILM guidance when using ElasticSearch for Workflow Instance and log

Open
#3,602 2 comments 1 reaction 0 assignees View on GitHub
triaged
Dominant language
C#
Stars
7.9k
Forks
1.5k
Avg merge
15h 22m
Merged PRs (30d)
114

Description

Regarding PR #3600

Elasticsearch is a distributed open search and analytics engine for all types of data. Very powerful for storing/indexing and then Searching Data store in JSON.

Data are stored in indices (which could be more or less be assimilate to Table in RDBMS Store, but it's not really that) and suffer of same issue when need to delete informations.
- in RDBMS, if you delete some record, some log events are created which growns the log size and slow the execution of deletion
- in ES, if you delete some record, some reindex need to occurs to indicate that record were deleted and this is not performant.

In RDBMS , the best solution is to use `TRUNCATE` but you'll loose all the data in the table (we cannot `TRUNCATE` with `WHERE` Statement).

in ES, the best solution is to delete Indice but you need to be sure that data could always be inserted. For this , ES has the notion of [Alias](https://www.elastic.co/guide/en/elasticsearch/reference/current/aliases.html). On client side, it's quite the same thing as a Index. On Server Side, the Alias will point to multiple indices for read and 1 index for write.

Regarding some temporale data , you can have :
- 1 alias named `WorkflowInstance`
- multiple indice named `WorkflowInstance_YYYYMM` where YYYY is year and MM is Month. The current month is the Writable(and readbable) index and others are readable indices from the alias. With this, if you need to purge some data, you can delete the index were MM is N-2 (for 2 month of history) without any downtime for reading/writing data from client side.

ES provide also some [Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html) which will manage for you the name of the index and the creation/deletion of indices regarding specific rules.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.