elastic / elastic/logstash

In logstash/util/charset.rb, we are not dealing with invalid UTF-8 well enough.

Open
#4,560 2 comments 0 reactions 1 assignee Claimed by @guyboertje View on GitHub
bug
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Particularly `ArgumentError: invalid byte sequence in UTF-8` errors.
- force_encoding does not fix invalid byte sequences
- we only encode with repair if the original charset is **not** utf-8
- we use an inspect side effect hack to escape bad characters

The inspect hack does not work with JSON strings:

``` ruby
> h = {"foo" => "bar"}
=> {"foo"=>"bar"}
> str = JrJackson::Json.dump(h, {})
=> "{\"foo\":\"bar\"}"
> str1 = str.inspect[1..-2]
=> "{\\\"foo\\\":\\\"bar\\\"}"
> JrJackson::Json.load(str1, {})
JrJackson::ParseError: Unexpected character ('\' (code 92)): was expecting double-quote
to start field name
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.