Joystream / Joystream/joystream
Discovery node sourcing data from the query node's database
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 116
- PR merge metrics
- No merged PRs in 30d
Description
The current design of the discovery node (https://github.com/Joystream/joystream/issues/324) assumes ingesting raw events directly from the joystream blockchain and then uploading to a downstream ElasticSearch cluster for indexing. It seems that exactly the same logic of replaying historical transactions is going to be implemented in the query node module, with the only difference that the blockchain state is kept and updated in the PostgresSQL. This suggests that replicating the blockchain state from PostgresSQL to ElasticSearch may be a more robust approach:
- Avoid duplicating the complex logic of replaying the transactions
- Ensure the schema consistency between the database and ES
- Nearly off-the-shelf solutions exist for such pipelines.
More concretely, Debezium specialises on a highly-scalable distribution of incremental database update through Kafka topics. This project suggests that a Debezium-based PostgreSQL -> ElasticSearch pipeline can be set up with rather low overheads.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The payload names no repository files, tests, or implementation entry points. Start by reading issue #324 and evaluating the proposed Debezium PostgreSQL-to-Elasticsearch pipeline; the issue does not define acceptance criteria for when the work is done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, kafka, postgresql, rust
- Domain
- blockchain, data-engineering, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100