fluent-bit out_http msgpack format is not compatible with fluentd in_http
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
I'm trying to send logs from fluentbit (0.13) via out_http to fluentd(v1.2.4)'s in_http source, it works with "format json" setting, but when I switched to "format msgpack", I'm getting this error:
```
500 Internal Server Error
undefined method `[]=' for 1533737116:Fixnum
Did you mean? []
```
the error is thrown here: https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_http.rb#L230
record got from fluentbit is so:
```
record=[2018-08-08 14:35:20.002109900 +0000, {"cpu_p"=>1.7500000000000002, "user_p"=>1.0, "system_p"=>0.75, "cpu0.p_cpu"=>0.0, "cpu0.p_user"=>0.0, "cpu0.p_system"=>0.0, "cpu1.p_cpu"=>5.0, "cpu1.p_user"=>3.0, "cpu1.p_system"=>2.0, "cpu2.p_cpu"=>1.0, "cpu2.p_user"=>0.0, "cpu2.p_system"=>1.0, "cpu3.p_cpu"=>1.0, "cpu3.p_user"=>1.0, "cpu3.p_system"=>0.0}]
```
if I choose "format json", the record looks like so:
```
record=[{"date"=>1533795317.001521, "cpu_p"=>2.25, "user_p"=>0.5, "system_p"=>1.75, "cpu0.p_cpu"=>1.0, "cpu0.p_user"=>0.0, "cpu0.p_system"=>1.0, "cpu1.p_cpu"=>6.0, "cpu1.p_user"=>1.0, "cpu1.p_system"=>5.0, "cpu2.p_cpu"=>1.0, "cpu2.p_user"=>0.0, "cpu2.p_system"=>1.0, "cpu3.p_cpu"=>1.0, "cpu3.p_user"=>1.0, "cpu3.p_system"=>0.0}]
```
as I understand from here: https://github.com/fluent/fluent-bit/blob/master/plugins/out_http/http.c#L435
fluentbit directly sends what it got from input plugin to fluentd if it's msgpack, but if it's json, it first converts msgpack to json, get first item in array as time, and injects it to record as date field.
because of this behavioural difference, in fluentd side it gets timestamp `2018-08-08 14:35:20.002109900 +0000` as single_record, and fails here: https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_http.rb#L204 with
`undefined method `[]=' for 1533737116:Fixnum`
it seems very fair to expect fluentbit out_http to work with fluentd in_http via msgpack format, but I couldn't find people complaining about the incompatibility. am I wrong? I'd be glad if somebody can verify the existence of this incompatibility.
since behavioural difference is caused by fluentbit, it seems the best thing to do is adding a new format type like msgpack_date_in_record in fluentbit side for out_http.
Contributor guide
Assessment
This issue has not been assessed yet.