openzim / openzim/node-libzim

Iterating a large archive leaks native memory: Blob/Item/Entry wrappers never report their size to V8

Open
#214 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
36
Forks
14
Avg merge
9d 1h
Merged PRs (30d)
1

Description

Environment: @openzim/libzim 4.4.0 (bundled libzim.so.9), Node v26.2.0, Linux x64

Reading content while iterating a large archive grows RSS without bound until the OS kills the process. The JS objects are collectable, but the native memory behind Blob (and Item/Entry) is never registered with V8 via Napi::MemoryManagement::AdjustExternalMemory. V8 only sees a tiny wrapper, so it never feels heap pressure, never schedules the finalizers, and the zim::Blob content buffers accumulate.

Reproduction: a single iterEfficient() pass over a ~50 GB wikipedia_en_all_nopic ZIM reading it.data.data per entry.

Observed: RSS climbs ~linearly (~48 KB/article here) into the tens of GB; it's RssAnon, not RssFile; the cluster cache is not the cause; gc() alone does nothing (finalizers queue on an event loop a sync loop never yields to) only gc() + await a turn holds it flat.

Root cause: Blob (src/blob.h) is a Napi::ObjectWrap whose blob_ owns blob_.size() bytes of content, created per read via Blob::New (src/item.h getData). Nothing in src/ calls AdjustExternalMemory and there's no destructor accounting for it.

Suggested fix: AdjustExternalMemory(env, +blob_.size()) on wrap, -size in a ~Blob() destructor. Then V8 tracks the real footprint and self-bounds. No caller-side --expose-gc.

Workaround: node --expose-gc + periodic gc()-then-await-a-turn.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/blob.h and the Blob::New path in src/item.h getData, then inspect how Napi::ObjectWrap instances are finalized. Add the external-memory accounting described in the issue and verify it with the large-archive iterEfficient() reproduction, checking that RSS remains bounded without caller-side --expose-gc.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, nodejs
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.