fluent / fluent/fluent-logger-ruby

encoding issue UnicodeDecodeError

Open
#86 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
256
Forks
77
PR merge metrics
No merged PRs in 30d

Description

Something within the logging pipeline is breaking encoding, but just for some characters.
I'm having a hard time reproducing this issue and i cannot pin point what is actually causing this but it seems that on the fluentd level, either the logger or fluentd itself.

I deployed fluentd in production and sending events from the Rails app using this logger. The logger is configured to send events to fluentd which sends it to S3 in as a gzip file.
I then have a processing pipeline using these files and here is where i started seeing the issues.

client config
```
client = Fluent::Logger::FluentLogger.new(
nil,
host: "localhost",
port: 24224,
use_nonblock: true,
wait_writeable: false
)
client.post("foo", event)
```
fluentd config
```

@type s3
@id S3_output

s3_bucket my-bucket
s3_region us-east-1

acl bucket-owner-full-control
store_as gzip_command

path preprocessed_logs/year=%Y/month=%-m/day=%-d/hour=%-H
s3_object_key_format "%{path}/#{Socket.gethostname}_%{hex_random}_%{index}.%{file_extension}"


timekey 300
timekey_use_utc true
timekey_wait 30
@type file
path /var/log/td-agent/buffer/foo


@type json

```

It seems that some characters are badly encoded.
here is this user agent example:
```
'Mozilla/5.0 (iPhone; CPU iPhone OS 13_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Clube da Fluência'
```
was logged as:
```
'Mozilla/5.0 (iPhone; CPU iPhone OS 13_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Clube da Flu\xeancia'
```

`ê` got changed to `\xea` which is breaking decoding.

Do you think this might be something to do with how the logger is sending data to fluentd?

to add more context, I'm using this logger in a Rails app and what I log is requests informations. I have checked the Rails side of things and the string passed to the logger is UTF-8 encoded.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.