crypto-com / crypto-com/chain-indexing

Problem: Current projection design is not scalable

Open
#622 1 comment 0 reactions 3 assignees Claimed by @davcrypto View on GitHub
Dominant language
Go
Stars
34
Forks
32
PR merge metrics
No merged PRs in 30d

Description

Currently, as Crypto.org Chain has more than 3 million blocks, it takes a long time for a newly indexing server to catch up the latest block height.

This will force us to spend a long time in:
- re-indexing existing projections
- deploying and testing newly introduced projections

One possible solution is to make use of `EVENT_STORE` mode. I think a more efficient way to catch up would be:

1. `EVENT_STORE` mode indexing server starts to fetch block data and write to DB.
2. Each projection creates multiple goroutines, to parallel handle these blocks.
3. After catching up to the latest height, each projection may only need one goroutine to continue working.

The difficulty here I think is mainly on Step 2, as for some projections, we have the assumption that the blocks come in an ordered manner. So to fulfill the parallel handling purpose, these projections might need refactoring.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.