crypto-org-chain / crypto-org-chain/cronos
Problem: memiavl has relatively big gc pressure
- Dominant language
- Go
- Stars
- 336
- Forks
- 299
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 4
Description
Depending on the traffic of the chain and interval of snapshot rewriting, memiavl need to manage lots of small memories for `MemNode`s and key/value slices, which could put a big pressure to golang GC, it might make sense to use some manual memory management.
## Memory Arenas
One obvious solution is [memory arenas](https://uptrace.dev/blog/golang-memory-arena.html) feature introduced in go 1.20, we just allocate these objects with big chunks of arenas.
### Negatives
- we can't reclaim and reuse memory, arenas don't support deallocating, before dropped as a whole when we switch snapshot, it's maybe alright assuming deletion happens more rarely than insertion/updates.
### Manual Memory Management
- Implement `MemNode` as a `[]byte`, similar to the nodes in mmap-ed snapshots, allocate in a big buffer, maintain an intrusive free list to reuse nodes.
- Leave key/value slices to golang GC.
Contributor guide
Research direction
Start by profiling memiavl under representative chain traffic and snapshot rewriting, focusing on MemNode and key/value slice allocations. Compare the proposed memory-arena approach with manual MemNode allocation and reuse, while leaving key/value slices to Go's GC. Done should include an evidence-backed implementation direction and measured GC impact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100