Node binding retains large XLSX RSS high-water per libuv worker
- Dominant language
- Rust
- Stars
- 21.5k
- Forks
- 1.3k
- Avg merge
- 42m
- Merged PRs (30d)
- 17
Description
## Summary
Sequential large-XLSX conversions through the Node binding retain roughly 135–140 MiB of RSS high-water **per libuv worker**. The process plateaus after every worker has handled one conversion, so this does not look like an unbounded leak, but the plateau scales almost linearly with `UV_THREADPOOL_SIZE`.
This reproduces in plain Node.js with no application imports.
## Environment
- `@firecrawl/anydoc`: 0.2.4
- Node.js: 24.18.0
- Linux x64, glibc 2.42
- Input: 20,000 rows × 6 columns, 1,411,780 compressed bytes
- Output: 824,747 Markdown characters
## Reproduction
Install `@firecrawl/anydoc@0.2.4` and `xlsx`, then save this as `repro.mjs`:
```js
import { toMarkdownBytes } from '@firecrawl/anydoc'
import * as XLSX from 'xlsx'
const data = [['Item', 'SKU', 'Brand', 'Qty', 'Unit', 'Marker']]
for (let index = 0; index < 20_000; index += 1) {
data.push([
`Industrial item ${index}`,
`SKU-${index}`,
'Brand',
index + 1,
'pcs',
`ITEM-${String(index).padStart(6, '0')}`,
])
}
const workbook = XLSX.utils.book_new()
XLSX.utils.book_append_sheet(workbook, XLSX.utils.aoa_to_sheet(data), 'Items')
const input = new Uint8Array(
XLSX.write(workbook, { type: 'buffer', bookType: 'xlsx', compression: true }),
)
for (let index = 0; index < 10; index += 1) {
const markdown = await toMarkdownBytes(input, null, { ocr: 'reject' })
console.log({
conversion: index + 1,
chars: markdown.length,
rssMiB: Math.round(process.memoryUsage().rss / 1024 / 1024 * 100) / 100,
})
}
```
Run the same sequential workload with different pool sizes:
```bash
UV_THREADPOOL_SIZE=1 node repro.mjs
UV_THREADPOOL_SIZE=2 node repro.mjs
UV_THREADPOOL_SIZE=4 node repro.mjs
UV_THREADPOOL_SIZE=8 node repro.mjs
```
## Observed steady-state RSS
| `UV_THREADPOOL_SIZE` | Workers | RSS after 10 sequential conversions |
|---:|---:|---:|
| 1 | 1 | 210.19 MiB |
| 2 | 2 | 345.41 MiB |
| 4 | 4 | 617.50 MiB |
| 8 | 8 | 1,151.24 MiB |
With the default pool of four workers, RSS rose once per conversion for the first four conversions:
```text
192.6 → 333.2 → 473.9 → 614.6 MiB
```
It then plateaued around 630–632 MiB.
The calls are strictly sequential and each Promise is awaited. JS heap stays small; the growth is native RSS.
Setting `MALLOC_ARENA_MAX=1` before process start collapses the eight-worker plateau from about 1,135 MiB to about 195 MiB on the same machine:
```bash
MALLOC_ARENA_MAX=1 UV_THREADPOOL_SIZE=8 node repro.mjs
```
## Likely boundary
The Node binding returns `napi::AsyncTask`, so sequential conversions can execute on different libuv workers. The results are consistent with the Rust/system allocator retaining the XLSX parser's high-water allocation in a separate glibc arena for each worker.
The XLSX parser necessarily creates sizeable intermediate state, but multiplying that high-water by every generic libuv worker makes process RSS depend on unrelated thread-pool configuration.
## Expected
After a sequential conversion completes and its result is no longer referenced, subsequent sequential conversions should reuse memory without multiplying the native RSS plateau by the number of libuv workers.
Possible solution boundaries might be a dedicated conversion pool, a synchronous binding that callers can isolate in their own worker, an allocator strategy that returns/reuses cross-thread memory, or documented/configurable mitigation. I do not want to prescribe which is safest for the project.
## Additional runtime confirmation
Bun 1.4.0 shows the same per-worker behavior through N-API. Its adaptive pool makes the effect more visible when other libraries have already expanded the pool, but Node alone is sufficient to reproduce the issue.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the provided repro.mjs and run the sequential conversion under several UV_THREADPOOL_SIZE values, including the MALLOC_ARENA_MAX comparison. Then inspect the Node binding entry point returning napi::AsyncTask. Done means a chosen mitigation is validated against the per-worker RSS plateau, with the behavior and any configuration requirements documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, rust
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100