Iterating a large archive leaks native memory: Blob/Item/Entry wrappers never report their size to V8
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 36
- Forks
- 14
- Avg merge
- 9d 1h
- Merged PRs (30d)
- 1
Description
Environment: @openzim/libzim 4.4.0 (bundled libzim.so.9), Node v26.2.0, Linux x64
Reading content while iterating a large archive grows RSS without bound until the OS kills the process. The JS objects are collectable, but the native memory behind Blob (and Item/Entry) is never registered with V8 via Napi::MemoryManagement::AdjustExternalMemory. V8 only sees a tiny wrapper, so it never feels heap pressure, never schedules the finalizers, and the zim::Blob content buffers accumulate.
Reproduction: a single iterEfficient() pass over a ~50 GB wikipedia_en_all_nopic ZIM reading it.data.data per entry.
Observed: RSS climbs ~linearly (~48 KB/article here) into the tens of GB; it's RssAnon, not RssFile; the cluster cache is not the cause; gc() alone does nothing (finalizers queue on an event loop a sync loop never yields to) only gc() + await a turn holds it flat.
Root cause: Blob (src/blob.h) is a Napi::ObjectWrap whose blob_ owns blob_.size() bytes of content, created per read via Blob::New (src/item.h getData). Nothing in src/ calls AdjustExternalMemory and there's no destructor accounting for it.
Suggested fix: AdjustExternalMemory(env, +blob_.size()) on wrap, -size in a ~Blob() destructor. Then V8 tracks the real footprint and self-bounds. No caller-side --expose-gc.
Workaround: node --expose-gc + periodic gc()-then-await-a-turn.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/blob.h and the Blob::New path in src/item.h getData, then inspect how Napi::ObjectWrap instances are finalized. Add the external-memory accounting described in the issue and verify it with the large-archive iterEfficient() reproduction, checking that RSS remains bounded without caller-side --expose-gc.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, nodejs
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100