Burrow fails to poll consumers with specific json format

Nobody has claimed this yet.

Hi,

Burrow in latest version fails to poll consumers which are Kafka Connect sink consumers when a specific json is request by the Sink connect.

**In Burrow logs, I can see the following warning messages:**

`burrow_1 | {"level":"warn","ts":1556442689.1351233,"msg":"failed to decode","type":"module","coordinator":"consumer","class":"kafka","name":"local","offset_topic":"__consumer_offsets","offset_partition":31,"offset_offset":0,"message_type":"metadata","group":"console-consumer-26002","reason":"value version","version":2}
burrow_1 | {"level":"warn","ts":1556442698.5125833,"msg":"failed to decode","type":"module","coordinator":"consumer","class":"kafka","name":"local","offset_topic":"__consumer_offsets","offset_partition":31,"offset_offset":1,"message_type":"metadata","group":"console-consumer-26002","reason":"value version","version":2}
burrow_1 | {"level":"warn","ts":1556442707.0164332,"msg":"failed to decode","type":"module","coordinator":"consumer","class":"kafka","name":"local","offset_topic":"__consumer_offsets","offset_partition":28,"offset_offset":0,"message_type":"metadata","group":"console-consumer-94257","reason":"value version","version":2}
burrow_1 | {"level":"warn","ts":1556442743.1726367,"msg":"failed to decode","type":"module","coordinator":"consumer","class":"kafka","name":"local","offset_topic":"__consumer_offsets","offset_partition":28,"offset_offset":1,"message_type":"metadata","group":"console-consumer-94257","reason":"value version","version":2}`

The messages look like the following:

`{"time":"1556442710","host":"ip-10-0-0-173","source":"splunk","sourcetype":"syslog:unassigned","index":"osnix","event":"Apr 28 09:11:50 ip-10-0-0-173 influxd[1392]: [httpd] 127.0.0.1 - - [28/Apr/2019:09:11:50 +0000] \"POST /write?db=telegraf HTTP/1.1\" 204 0 \"-\" \"Telegraf/1.9.3\" a7eee047-6995-11e9-8bfa-000000000000 24794","fields": {"input":"imtcp_shared","rawmsg":"<30>Apr 28 09:11:50 ip-10-0-0-173 influxd[1392]: [httpd] 127.0.0.1 - - [28/Apr/2019:09:11:50 +0000] \"POST /write?db=telegraf HTTP/1.1\" 204 0 \"-\" \"Telegraf/1.9.3\" a7eee047-6995-11e9-8bfa-000000000000 24794","severity":"info","facility":"daemon","protocol-version":"0","app-name":"influxd","procid":"1392","msgid":"-","structured-data":"-","program-name":"influxd"}}
{"time":"1556442720","host":"ip-10-0-0-173","source":"splunk","sourcetype":"syslog:unassigned","index":"osnix","event":"Apr 28 09:12:00 ip-10-0-0-173 influxd[1392]: [httpd] 127.0.0.1 - - [28/Apr/2019:09:12:00 +0000] \"POST /write?db=telegraf HTTP/1.1\" 204 0 \"-\" \"Telegraf/1.9.3\" ade4c4e7-6995-11e9-8bfb-000000000000 205273","fields": {"input":"imtcp_shared","rawmsg":"<30>Apr 28 09:12:00 ip-10-0-0-173 influxd[1392]: [httpd] 127.0.0.1 - - [28/Apr/2019:09:12:00 +0000] \"POST /write?db=telegraf HTTP/1.1\" 204 0 \"-\" \"Telegraf/1.9.3\" ade4c4e7-6995-11e9-8bfb-000000000000 205273","severity":"info","facility":"daemon","protocol-version":"0","app-name":"influxd","procid":"1392","msgid":"-","structured-data":"-","program-name":"influxd"}}
{"time":"1556442730","host":"ip-10-0-0-173","source":"splunk","sourcetype":"syslog:unassigned","index":"osnix","event":"Apr 28 09:12:10 ip-10-0-0-173 influxd[1392]: [httpd] 127.0.0.1 - - [28/Apr/2019:09:12:10 +0000] \"POST /write?db=telegraf HTTP/1.1\" 204 0 \"-\" \"Telegraf/1.9.3\" b3db4514-6995-11e9-8bfc-000000000000 55844","fields": {"input":"imtcp_shared","rawmsg":"<30>Apr 28 09:12:10 ip-10-0-0-173 influxd[1392]: [httpd] 127.0.0.1 - - [28/Apr/2019:09:12:10 +0000] \"POST /write?db=telegraf HTTP/1.1\" 204 0 \"-\" \"Telegraf/1.9.3\" b3db4514-6995-11e9-8bfc-000000000000 55844","severity":"info","facility":"daemon","protocol-version":"0","app-name":"influxd","procid":"1392","msgid":"-","structured-data":"-","program-name":"influxd"}}
{"time":"1556442740","host":"ip-10-0-0-173","source":"splunk","sourcetype":"syslog:unassigned","index":"osnix","event":"Apr 28 09:12:20 ip-10-0-0-173 influxd[1392]: [httpd] 127.0.0.1 - - [28/Apr/2019:09:12:20 +0000] \"POST /write?db=telegraf HTTP/1.1\" 204 0 \"-\" \"Telegraf/1.9.3\" b9d085d7-6995-11e9-8bfd-000000000000 54541","fields": {"input":"imtcp_shared","rawmsg":"<30>Apr 28 09:12:20 ip-10-0-0-173 influxd[1392]: [httpd] 127.0.0.1 - - [28/Apr/2019:09:12:20 +0000] \"POST /write?db=telegraf HTTP/1.1\" 204 0 \"-\" \"Telegraf/1.9.3\" b9d085d7-6995-11e9-8bfd-000000000000 54541","severity":"info","facility":"daemon","protocol-version":"0","app-name":"influxd","procid":"1392","msgid":"-","structured-data":"-","program-name":"influxd"}}
`

**The SInk consumer is the Splunk Connect for Kafka connector:**

https://github.com/splunk/kafka-connect-splunk

**If the connector uses an internal option:**

`"splunk.hec.json.event.formatted" : "true"`

Which tells the connector that the messages are in a specific format, then Burrow fails to poll the consumer with the message shown above.

**If the consumer does not use this option, then Burrow successfully polls the consumer lagging with no problem. (but then the data is not ingested properly on the consumer side when this option is required)**

I cannot figure out why activating the option leads Burrow not to be able to poll the consumer...

Thank you for your help!

Guilhem

Reproduce the consumer-polling failure using the Splunk Connect for Kafka connector with "splunk.hec.json.event.formatted" set to true, and compare it with the working configuration. Trace Burrow's consumer metadata decoding path from the warning showing value version 2; done means Burrow successfully polls and reports lag for the formatted consumer.

Written by the indexing model from the issue text.