electric-sql / electric-sql/electric
Reduce memory usage footprint
- Dominant language
- TypeScript
- Stars
- 10.4k
- Forks
- 375
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 18
Description
We still have a large memory footprint for each shape. We think this comes memory not all memory allocated reclaimed during the lifetime of a consumer.
With the new process shutdown approach (https://github.com/electric-sql/electric/pull/3455), we expect improvements as processes recycle more often. We should observe that, otherwise need to dig further.
The next thing would be to enforce a cap on memory usage, since larger workloads might have more "active" shapes. The idea would need to enforce a max number of consumers at any point. The challenge is that transaction handling must be resilient to shapes being opened/closed multiple times during the handling of a transactions, as large transactions may touch many rows.
Alternative approach: Replace ETS with SQLite for automatic memory management via disk offloading. Use a single SQLite per stack - previous benchmarks showed comparable performance to ETS.
Contributor guide
Assessment
This issue has not been assessed yet.