Joystream / Joystream/joystream
Minimise use and processing of large arrays
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 116
- PR merge metrics
- No merged PRs in 30d
Description
We previously identified cpu intensive tasks processing large arrays, in particular the `lodash` `differenceWith` method, comparing two very large arrays. `intersection` used in `getLocalDataObjectsByBagId()` state api endpoint could also be problematic.
We should generally avoid this.
Choosing `Map` or `Set` if we are storing the data long term in memory.
For processing, try to fetch data in chunks, (gql queries with paging and result set limits). Using Async generators to make programming around this approach more efficient.
Another place where we "produce" large arrays, is with `fs.promises.readdir()` when reading list of objects in the uploads folder. Its okay to do it on startup, but look for places where we might do it more frequently like in the state api endpoint: `getLocalDataStats()`
Some examples in https://github.com/Joystream/joystream/pull/5026
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.