firecrawl / firecrawl/anydoc

OOXML: recoverable allocation failure in Package::part is classified as malformed

Open
#167 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21.5k
Forks
1.3k
Avg merge
42m
Merged PRs (30d)
17

Description

## Summary

In `@firecrawl/anydoc@0.2.4`, a recoverable allocation failure while reading an OOXML ZIP part can be reported as `code: "malformed"` with:

```text
malformed document (word/document.xml): corrupt archive entry: out of memory
```

The memory pressure here is deliberately imposed by our subprocess sandbox. This report concerns the error classification when an OOM reaches the error-return path, not a requirement that every native allocation failure be recoverable or that conversion succeed under the limit.

## Environment and observations

- Bun `1.4.1+4661e494f`, Linux x64 GNU, native AnyDoc npm binding `0.2.4`.
- Fresh child per conversion; empty environment; `--smol --no-env-file --config=/dev/null`; input through stdin.
- `RLIMIT_DATA` soft/hard: 536870912 bytes; core dumps disabled; outer GNU `timeout` of 5 seconds. This is an allocation limit, not an RSS cap or an observed kernel OOM kill.
- Synthetic DOCX: 73,241 compressed bytes; `word/document.xml` is 67,109,048 bytes and contains one run with 64 MiB of ASCII `Z` text. The ZIP entries other than that XML part are retained from a valid DOCX.
- Across 120 previously recorded runs of the same input: 117 children terminated with SIGABRT; 3 reached the JS catch with the exact error above (`code: "malformed"`). Stdout was empty in every run. The three caught cases were run indices 2, 93 and 104 (zero-based). This is an observed distribution, not a guaranteed reproduction rate.

The same input is recognized as `docx` before conversion. The unusual error is specifically `out of memory`, not a checksum or XML syntax failure.

## Reproduction setup

Install `@firecrawl/anydoc@0.2.4` in an isolated directory with Bun 1.4.1. Prepare the stress input from any valid `base.docx`:

```python
from zipfile import ZipFile, ZIP_DEFLATED

prefix = b''
suffix = b''
with ZipFile('base.docx') as source, ZipFile('expanded.docx', 'w', ZIP_DEFLATED) as output:
for entry in source.infolist():
data = prefix + b'Z' * (64 * 1024 * 1024) + suffix if entry.filename == 'word/document.xml' else source.read(entry)
output.writestr(entry.filename, data)
```

The original measured fixture's SHA-256 is `9ebc7b1eed5c4ae34863dad3af2da274c57ecd888b34dc9360682536a499153b`; a different base document/ZIP metadata will produce different bytes and may change the allocation outcome.

Use a child that reads stdin, calls `formatFromBytes(bytes)`, then `await toMarkdownBytes(bytes, format, { ocr: 'reject' })`, and logs `error.code` and `error.message` from its catch. Run each child with an absolute Bun path:

```sh
env -i /usr/bin/timeout --foreground --signal=KILL --kill-after=0.1s 5s \
/usr/bin/prlimit --data=536870912:536870912 --core=0:0 -- \
/absolute/path/to/bun --smol --no-env-file --config=/dev/null ./child.ts \
< expanded.docx
```

Allocation timing/reservations affect whether the process aborts or returns an error; one run may only show SIGABRT. The 120-run observation used an application stdin reader before the AnyDoc call, so the simplified harness above is a reproduction setup, not a claim that it reproduces the same frequency.

## Source evidence and expected result

[`Package::part` in v0.2.4](https://github.com/firecrawl/anydoc/blob/v0.2.4/src/package/archive.rs#L63-L68) maps every `read_to_end` error to `ConvertError::Malformed` and prefixes it with `corrupt archive entry`. The same mapping is present in [current main at 261fc257](https://github.com/firecrawl/anydoc/blob/261fc257d17c3eab0f673be31c408fd9fdc2171a/src/package/archive.rs#L63-L68) (source inspection; only 0.2.4 was run).

When a recoverable `std::io::ErrorKind::OutOfMemory` is returned, preserve its allocation/resource-failure meaning instead of classifying the document as corrupt. `ResourceLimit` or another distinct error category would let callers distinguish resource exhaustion from malformed input without matching message text. Ordinary corrupt ZIP errors should remain `Malformed`.

A focused regression could inject an `OutOfMemory` read error and a genuine corrupt-input error, verifying that their categories differ. This does not require an allocator-dependent stress test to return the caught error every time.

Our current workaround recognizes only `code === 'malformed'` together with the exact `: corrupt archive entry: out of memory` suffix; native aborts remain separate subprocess failures.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/package/archive.rs at Package::part and inspect how read_to_end errors become ConvertError categories. Add focused coverage for an injected OutOfMemory read error and a genuine corrupt-input error, then run the relevant package tests. Done means resource exhaustion has its own category while ordinary corrupt ZIP errors remain Malformed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.