Joystream / Joystream/joystream

QN Indexer Performance

Open
#5,212 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
116
PR merge metrics
No merged PRs in 30d

Description

follows #3748 #4353 #4724

## Rationale

Next to querying performance (#5088) we also need to track indexing. Our QN (https://query.l1.media/graphql) is up for 1 month and synced to block 8.8M so far.

The recent increase of the [WORKERS_NUMBER](https://github.com/Joystream/joystream/blob/master/docker-compose.yml#L248) for the [indexer](https://github.com/Joystream/hydra) container from 5 to 15 did not affect process load (11 Threads, 130% load, one core only) but visibly increases the pressure on postgres. Does around 1.6k blocks per minute, that's still 64h for the remaining 6M blocks.

The transition to Subsquid (#2893 Joystream/atlas#1811 joystream/pioneer#2002) was canceled by JSG, presumably due to lack of funds. Please add specifics about the reasoning.

## Profiling
Proposal for further profiling to identify and optimize most expensive functions. Steps:
1. Run the [indexer](https://github.com/Joystream/hydra/blob/master/hydra-indexer.md) standalone or as container (joystream/hydra-indexer:v5.0.0-alpha.8): `docker exec -t indexer bash`
2. Modify the command line to:
```
/usr/local/bin/node --prof ./lib/run.js index
```
Or use other profiling tools (eg. benchmark).

## Apps note
Since the indexer gateway vanished (Joystream/orion/issues/371) provide a public alternative for atlas operators.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.