In logstash/util/charset.rb, we are not dealing with invalid UTF-8 well enough.
Open
bug
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
Particularly `ArgumentError: invalid byte sequence in UTF-8` errors.
- force_encoding does not fix invalid byte sequences
- we only encode with repair if the original charset is **not** utf-8
- we use an inspect side effect hack to escape bad characters
The inspect hack does not work with JSON strings:
``` ruby
> h = {"foo" => "bar"}
=> {"foo"=>"bar"}
> str = JrJackson::Json.dump(h, {})
=> "{\"foo\":\"bar\"}"
> str1 = str.inspect[1..-2]
=> "{\\\"foo\\\":\\\"bar\\\"}"
> JrJackson::Json.load(str1, {})
JrJackson::ParseError: Unexpected character ('\' (code 92)): was expecting double-quote
to start field name
```
Contributor guide
Assessment
This issue has not been assessed yet.