3b1b / 3b1b/captions

Incorrect newline characters breaking JSON parsing

Open
#366 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
286
Forks
151
PR merge metrics
No merged PRs in 30d

Description

See https://github.com/3b1b/captions/blob/main/2023/gaussian-integral/hebrew/sentence_translations.json#L774

I think this is the AI model trying to translate a `\n` newline character, and using a Hebrew "n" instead, which is not a valid JSON escape character. So, parsing fails, and going to that lesson page shows that the captions file is missing (I could improve the message to discriminate between loading errors and parsing errors).

It'd be hard to make the app recover from this type of parsing error though. I could replace all `\מ`s with `\n`s, but what about other languages and escape characters? Perhaps a better solution here would be to make sure these characters are removed from the input English _before_ passing them to the models. Could more easily make sure all escape characters are captured that way.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue points to a specific JSON file (2023/gaussian-integral/hebrew/sentence_translations.json) where a Hebrew character is breaking parsing. Look at how caption files are loaded and parsed in the codebase. The task is to find where English text is sent to translation models and ensure escape characters like '\n' are preserved or sanitized before translation. Check for existing validation or sanitization logic. A fix would involve updating the pre-processing step and verifying the JSON file parses correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
json, typescript
Domain
content, internationalization
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.