Joystream / Joystream/joystream

Discovery node sourcing data from the query node's database

Open
#406 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
1.4k
Forks
116
PR merge metrics
No merged PRs in 30d

Description

The current design of the discovery node (https://github.com/Joystream/joystream/issues/324) assumes ingesting raw events directly from the joystream blockchain and then uploading to a downstream ElasticSearch cluster for indexing. It seems that exactly the same logic of replaying historical transactions is going to be implemented in the query node module, with the only difference that the blockchain state is kept and updated in the PostgresSQL. This suggests that replicating the blockchain state from PostgresSQL to ElasticSearch may be a more robust approach:

  • Avoid duplicating the complex logic of replaying the transactions
  • Ensure the schema consistency between the database and ES
  • Nearly off-the-shelf solutions exist for such pipelines.

More concretely, Debezium specialises on a highly-scalable distribution of incremental database update through Kafka topics. This project suggests that a Debezium-based PostgreSQL -> ElasticSearch pipeline can be set up with rather low overheads.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The payload names no repository files, tests, or implementation entry points. Start by reading issue #324 and evaluating the proposed Debezium PostgreSQL-to-Elasticsearch pipeline; the issue does not define acceptance criteria for when the work is done.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, kafka, postgresql, rust
Domain
blockchain, data-engineering, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.