crypto-org-chain / crypto-org-chain/cronos

Problem: memiavl has relatively big gc pressure

Open
#988 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
336
Forks
299
Avg merge
2d 17h
Merged PRs (30d)
4

Description

Depending on the traffic of the chain and interval of snapshot rewriting, memiavl need to manage lots of small memories for `MemNode`s and key/value slices, which could put a big pressure to golang GC, it might make sense to use some manual memory management.

## Memory Arenas
One obvious solution is [memory arenas](https://uptrace.dev/blog/golang-memory-arena.html) feature introduced in go 1.20, we just allocate these objects with big chunks of arenas.

### Negatives

- we can't reclaim and reuse memory, arenas don't support deallocating, before dropped as a whole when we switch snapshot, it's maybe alright assuming deletion happens more rarely than insertion/updates.

### Manual Memory Management

- Implement `MemNode` as a `[]byte`, similar to the nodes in mmap-ed snapshots, allocate in a big buffer, maintain an intrusive free list to reuse nodes.
- Leave key/value slices to golang GC.

Contributor guide

Open the contributing guide

Research direction

Start by profiling memiavl under representative chain traffic and snapshot rewriting, focusing on MemNode and key/value slice allocations. Compare the proposed memory-arena approach with manual MemNode allocation and reuse, while leaving key/value slices to Go's GC. Done should include an evidence-backed implementation direction and measured GC impact.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.