elastic / elastic/logstash

UTF-8 in Windows

Open
#7,988 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Hello,

I posted this in discuss.elastic.co ([link](https://discuss.elastic.co/t/utf-8-in-windows/96092)) but I guess here is more appropriate.

There seems to be a problem with how UTF-8 input is handled in Logstash (5.5.0) on Windows (2008 R2/2012 R2); using the below config, any arabic input comes out as question marks `??????????`, and this seems independent of the input plugin (I tried ‘file’ and ‘beats’ inputs) and codecs (JSON/PLAIN).

```
input {
file {
path => "C:\ELK\temp\input.txt"
}
}
output {
file {
path => "C:\ELK\temp\output.txt"
}
}
```

Using `{charset => ["CP1252"]}` as proposed in [this](https://discuss.elastic.co/t/support-hebrew-characters-in-logs/81168) discussion does fix the issue, even though the input is UTF-8.

Strangely, the above config works as expected in Linux without specifying the CP1252 ‘charset’!!!

I did some testing and it seems this issue was introduced in v5.0.0, it worked as expected in v2.4.0.

To reproduce, add an arabic keyboard in Windows and just mash the keyboard :) , make sure that the input file is UTF-8 encoded (I use Notepad++).

Any thoughts on this are appreciated.

Thanks

Contributor guide

Open the contributing guide

Research direction

Reproduce the UTF-8 Arabic input on Windows with the shown file-input and file-output configuration, then compare the behavior with Linux and Logstash 2.4.0. Trace charset handling in the file and beats inputs and JSON/plain codecs; done means Arabic UTF-8 is preserved without forcing CP1252 and the regression is covered by a test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.