firecrawl / firecrawl/pdf-inspector

acroform field names and values are decoded as utf-8, mangling utf-16be text strings

Open Beginner friendly
#504 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
19.1k
Forks
1.3k
Avg merge
9h 21m
Merged PRs (30d)
51

Description

form field names and values are decoded with `String::from_utf8_lossy`, but per PDF 32000-1 §7.9.2.2 a text string is UTF-16BE when it starts with a `FE FF` BOM and PDFDocEncoding otherwise. neither survives a utf-8 decode, so anything acrobat wrote comes back full of replacement chars and interleaved nulls.

hit this on the irs w-9, where all 27 field names are utf-16be. every one is mangled.

sites in `src/extractor/links.rs` (main @ 636ca1a5):

- 301, `/T` field name
- 374, `/V` string value
- 385, `/V` array member

lines 153 and 404 also call `from_utf8_lossy` but those are a `/URI` and a `Btn` name object, neither of which is a text string, so they look right to me as-is.

repro, dropped into the existing `mod tests` in links.rs:

```rust
fn utf16be(text: &str) -> Vec {
let mut b = vec![0xFE, 0xFF];
for u in text.encode_utf16() { b.extend_from_slice(&u.to_be_bytes()); }
b
}

#[test]
fn utf16be_field_name_and_value_decode() {
let mut doc = Document::new();
let widget = doc.add_object(dictionary! {
"Type" => "Annot", "Subtype" => "Widget", "FT" => "Tx",
"T" => Object::String(utf16be("Prénom"), lopdf::StringFormat::Literal),
"V" => Object::String(utf16be("Zoë"), lopdf::StringFormat::Literal),
"Rect" => vec![10.into(), 20.into(), 110.into(), 40.into()],
});
let page = doc.add_object(dictionary! { "Type" => "Page", "Annots" => vec![Object::Reference(widget)] });
let catalog = doc.add_object(dictionary! {
"Type" => "Catalog",
"AcroForm" => dictionary! { "Fields" => vec![Object::Reference(widget)] },
});
doc.trailer.set("Root", Object::Reference(catalog));

let items = extract_form_fields(&doc, &HashMap::from([(page, 1)]));
assert_eq!(items[0].text, "Prénom: Zoë");
}
```

gives:

```
left: "\u{fffd}\u{fffd}\0P\0r\0\u{fffd}\0n\0o\0m: \u{fffd}\u{fffd}\0Z\0o\0\u{fffd}"
right: "Prénom: Zoë"
```

the decoder is already in the tree and already used by `structure_tree.rs:1067` and `content_stream.rs:949`, links.rs just never calls it. `lopdf::decode_text_string` is probably the better of the two since it does real PDFDocEncoding rather than latin-1, which matters in the 0x80-0xA0 range (0xA0 is `€`, not nbsp).

happy to send a PR if useful, it's a three line change plus tests for both branches.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/extractor/links.rs at the `/T` and `/V` handling sites around lines 301, 374, and 385, then review the existing text decoder uses in structure_tree.rs and content_stream.rs. Extend the existing links.rs tests with the provided UTF-16BE case and a PDFDocEncoding case; done means field names and values decode correctly without changing `/URI` or `Btn` handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.