Joystream / Joystream/joystream

Minimise use and processing of large arrays

Open
#5,051 1 comment 0 reactions 0 assignees View on GitHub
colossus
Dominant language
Rust
Stars
1.4k
Forks
116
PR merge metrics
No merged PRs in 30d

Description

We previously identified cpu intensive tasks processing large arrays, in particular the `lodash` `differenceWith` method, comparing two very large arrays. `intersection` used in `getLocalDataObjectsByBagId()` state api endpoint could also be problematic.

We should generally avoid this.

Choosing `Map` or `Set` if we are storing the data long term in memory.

For processing, try to fetch data in chunks, (gql queries with paging and result set limits). Using Async generators to make programming around this approach more efficient.

Another place where we "produce" large arrays, is with `fs.promises.readdir()` when reading list of objects in the uploads folder. Its okay to do it on startup, but look for places where we might do it more frequently like in the state api endpoint: `getLocalDataStats()`

Some examples in https://github.com/Joystream/joystream/pull/5026

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.