UTF-8 in Windows
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
Hello,
I posted this in discuss.elastic.co ([link](https://discuss.elastic.co/t/utf-8-in-windows/96092)) but I guess here is more appropriate.
There seems to be a problem with how UTF-8 input is handled in Logstash (5.5.0) on Windows (2008 R2/2012 R2); using the below config, any arabic input comes out as question marks `??????????`, and this seems independent of the input plugin (I tried ‘file’ and ‘beats’ inputs) and codecs (JSON/PLAIN).
```
input {
file {
path => "C:\ELK\temp\input.txt"
}
}
output {
file {
path => "C:\ELK\temp\output.txt"
}
}
```
Using `{charset => ["CP1252"]}` as proposed in [this](https://discuss.elastic.co/t/support-hebrew-characters-in-logs/81168) discussion does fix the issue, even though the input is UTF-8.
Strangely, the above config works as expected in Linux without specifying the CP1252 ‘charset’!!!
I did some testing and it seems this issue was introduced in v5.0.0, it worked as expected in v2.4.0.
To reproduce, add an arabic keyboard in Windows and just mash the keyboard :) , make sure that the input file is UTF-8 encoded (I use Notepad++).
Any thoughts on this are appreciated.
Thanks
Contributor guide
Research direction
Reproduce the UTF-8 Arabic input on Windows with the shown file-input and file-output configuration, then compare the behavior with Linux and Logstash 2.4.0. Trace charset handling in the file and beats inputs and JSON/plain codecs; done means Arabic UTF-8 is preserved without forcing CP1252 and the regression is covered by a test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100