elsa-workflows / elsa-workflows/elsa-core
Use Alias and ILM guidance when using ElasticSearch for Workflow Instance and log
- Dominant language
- C#
- Stars
- 7.9k
- Forks
- 1.5k
- Avg merge
- 15h 22m
- Merged PRs (30d)
- 114
Description
Regarding PR #3600
Elasticsearch is a distributed open search and analytics engine for all types of data. Very powerful for storing/indexing and then Searching Data store in JSON.
Data are stored in indices (which could be more or less be assimilate to Table in RDBMS Store, but it's not really that) and suffer of same issue when need to delete informations.
- in RDBMS, if you delete some record, some log events are created which growns the log size and slow the execution of deletion
- in ES, if you delete some record, some reindex need to occurs to indicate that record were deleted and this is not performant.
In RDBMS , the best solution is to use `TRUNCATE` but you'll loose all the data in the table (we cannot `TRUNCATE` with `WHERE` Statement).
in ES, the best solution is to delete Indice but you need to be sure that data could always be inserted. For this , ES has the notion of [Alias](https://www.elastic.co/guide/en/elasticsearch/reference/current/aliases.html). On client side, it's quite the same thing as a Index. On Server Side, the Alias will point to multiple indices for read and 1 index for write.
Regarding some temporale data , you can have :
- 1 alias named `WorkflowInstance`
- multiple indice named `WorkflowInstance_YYYYMM` where YYYY is year and MM is Month. The current month is the Writable(and readbable) index and others are readable indices from the alias. With this, if you need to purge some data, you can delete the index were MM is N-2 (for 2 month of history) without any downtime for reading/writing data from client side.
ES provide also some [Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html) which will manage for you the name of the index and the creation/deletion of indices regarding specific rules.
Contributor guide
Assessment
This issue has not been assessed yet.