ruby / ruby/psych

SyntaxError for files with wrong encoding

Open
#517 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
597
Forks
223
Avg merge
11h 23m
Merged PRs (30d)
3

Description

I have multiple files that just look fine with editors (gedit, sublime, vim) but produce the following error message when loaded by Psych:

Psych.safe_load(file)
/usr/local/lib/ruby/3.0.0/psych.rb:456:in `parse': (<unknown>): invalid trailing UTF-8 octet at line 1 column 1 (Psych::SyntaxError)

A file example can be found here (renamed as attaching a .yaml file is not supported). I am not that familiar with character encoding, but the file seems to get interpreted as UTF-8 while it actually isn't. isutf8 returns metal_copper (1).yaml: line 87, char 12, byte 1497: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF. on the file example.

I do not know how editor software today handle this topic, but it sure would be desirable to have Psych parse files in the way as shown in common editors. From googling the error message it seems I am not the only one who had problems with that and it seem to produce problem now and then for users/project that use Psych.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at Psych.safe_load and the parse call reported at psych.rb:456, then reproduce the failure with the attached example_file.txt and inspect its encoding. The issue does not name a test or precise acceptance condition; completion would require an agreed behavior for non-UTF-8 input and a regression check for that example.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.