Joystream / Joystream/joystream
QN Indexer Performance
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 116
- PR merge metrics
- No merged PRs in 30d
Description
follows #3748 #4353 #4724
## Rationale
Next to querying performance (#5088) we also need to track indexing. Our QN (https://query.l1.media/graphql) is up for 1 month and synced to block 8.8M so far.
The recent increase of the [WORKERS_NUMBER](https://github.com/Joystream/joystream/blob/master/docker-compose.yml#L248) for the [indexer](https://github.com/Joystream/hydra) container from 5 to 15 did not affect process load (11 Threads, 130% load, one core only) but visibly increases the pressure on postgres. Does around 1.6k blocks per minute, that's still 64h for the remaining 6M blocks.
The transition to Subsquid (#2893 Joystream/atlas#1811 joystream/pioneer#2002) was canceled by JSG, presumably due to lack of funds. Please add specifics about the reasoning.
## Profiling
Proposal for further profiling to identify and optimize most expensive functions. Steps:
1. Run the [indexer](https://github.com/Joystream/hydra/blob/master/hydra-indexer.md) standalone or as container (joystream/hydra-indexer:v5.0.0-alpha.8): `docker exec -t indexer bash`
2. Modify the command line to:
```
/usr/local/bin/node --prof ./lib/run.js index
```
Or use other profiling tools (eg. benchmark).
## Apps note
Since the indexer gateway vanished (Joystream/orion/issues/371) provide a public alternative for atlas operators.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.