electric-sql / electric-sql/electric

Reduce memory usage footprint

Open
#3,490 2 comments 0 reactions 1 assignee Claimed by @magnetised View on GitHub
Dominant language
TypeScript
Stars
10.4k
Forks
375
Avg merge
3d 1h
Merged PRs (30d)
18

Description

We still have a large memory footprint for each shape. We think this comes memory not all memory allocated reclaimed during the lifetime of a consumer.

With the new process shutdown approach (https://github.com/electric-sql/electric/pull/3455), we expect improvements as processes recycle more often. We should observe that, otherwise need to dig further.

The next thing would be to enforce a cap on memory usage, since larger workloads might have more "active" shapes. The idea would need to enforce a max number of consumers at any point. The challenge is that transaction handling must be resilient to shapes being opened/closed multiple times during the handling of a transactions, as large transactions may touch many rows.

Alternative approach: Replace ETS with SQLite for automatic memory management via disk offloading. Use a single SQLite per stack - previous benchmarks showed comparable performance to ETS.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.