Parsing of long string literals with escaped characters is slow
Nobody has claimed this yet.
- Dominant language
- OCaml
- Stars
- 46
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
If you try to parse a long string value with a lot of escape sequences, the parser uses `String.sub` for getting 2-character substring like `\"` from the long string. Unfortunately, JavaScript implementation of `String.sub` works the way it converts the WHOLE string to `bytes` (one byte at a time), gets requested range from the bytes array and converts it back to string.
This necessarily means that if you have 200,000 char JSON string with 10,000 escape sequences, the whole string is traversed 10,000x :slightly\_frowning\_face:
https://github.com/jaredly/rex-json/blob/c929cb4cad483eb6c1be87eeeb50a58c360af1d3/src/Json.re#L274
**Solutions I've come up with:**
- Make sure that `Js.String.substrAtMost` is used when targeting JavaScript
- Or do you expect the OCaml-way of handling strings? (String.sub working on bytes instead of UTF8 characters)
- Usage: `Js.String.substrAtMost(~from=i, ~length=2, text)`
- Create or use some existing library for UTF8 strings that compiles to Js.String in case of JavaScript.
- Do you know some? I've just started with Reason/BS/OCaml so I'm not really sure if there is some "industry standard" of handling UTF strings in Ocaml...
Otherwise thank you for making this fantastically ergonomic library, the infix operators are great!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at src/Json.re around line 274 and inspect how String.sub is used while parsing escaped characters. Compare the JavaScript-specific substring option described in the issue, then verify that long strings with many escapes no longer repeatedly traverse the full input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, ocaml
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100