Bug: failed loki insert
Open
Nobody has claimed this yet.
bug
- Dominant language
- Go
- Stars
- 504
- Forks
- 263
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 5
Description
Describe the bug
if log line is json the insertion would fail
{"level":"error","ts":"2025-09-28T12:21:44.721518Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:38660}]]}]}|..., bigger context ...|tocol\":\"HTTP/2.0\",\"status\":\"304\",\"bytes\":38660}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.722507Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:17003}]]}]}|..., bigger context ...|tocol\":\"HTTP/2.0\",\"status\":\"503\",\"bytes\":17003}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.72251Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|\":2521}]]}]}|..., bigger context ...|otocol\":\"HTTP/1.0\",\"status\":\"410\",\"bytes\":2521}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.72286Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:26831}]]}]}|..., bigger context ...|tocol\":\"HTTP/1.1\",\"status\":\"200\",\"bytes\":26831}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.725175Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:42978}]]}]}|..., bigger context ...|tocol\":\"HTTP/1.0\",\"status\":\"404\",\"bytes\":42978}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.725769Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:41531}]]}]}|..., bigger context ...|tocol\":\"HTTP/1.0\",\"status\":\"301\",\"bytes\":41531}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.985293Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:22297}]]}]}|..., bigger context ...|\"HTTP/1.1\",\"status\":\"300\",\"bytes\":22297}]]}]}|...\n"}
To Reproduce
Steps to reproduce the behavior:
- file.d --config=./config.yaml
- config.yaml
pipelines:
loki:
settings:
capacity: 1000
input:
type: file
persistence_mode: async
# offsets_op: reset
paths:
include:
- '/Users/romanchechyotkin/file.d/demo_logs.json'
offsets_file: ./offsets.yaml
actions:
- type: json_decode
field: message
- type: set_time
field: timestamp
override: false
- type: convert_date
field: timestamp
target_format: unixtimenano
source_formats:
- 'rfc3339'
- 'rfc3339nano'
- '02/Jan/2006:15:04:05 -0700'
- '2006-01-02T15:04:05.999Z0700'
output:
type: loki
fatal_on_failed_insert: true
retry: 3
address: http://localhost:3100
timestamp_field: timestamp
message_field: message
- demo_logs.json
{"host":"localhost","message":"{\"host\":\"143.61.119.90\",\"user-identifier\":\"jesseddy\",\"datetime\":\"22/Sep/2025:17:25:23\",\"method\":\"HEAD\",\"request\":\"/user/booperbot124\",\"protocol\":\"HTTP/1.0\",\"status\":\"404\",\"bytes\":14476,\"referer\":\"https://we.bauhaus/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:23.842787109Z"}
{"host":"localhost","message":"{\"host\":\"53.108.32.5\",\"user-identifier\":\"KarimMove\",\"datetime\":\"22/Sep/2025:17:25:24\",\"method\":\"DELETE\",\"request\":\"/wp-admin\",\"protocol\":\"HTTP/1.1\",\"status\":\"503\",\"bytes\":28124,\"referer\":\"https://random.xn--ses554g/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:24.842790741Z"}
{"host":"localhost","message":"{\"host\":\"84.243.110.238\",\"user-identifier\":\"jesseddy\",\"datetime\":\"22/Sep/2025:17:25:25\",\"method\":\"PATCH\",\"request\":\"/secret-info/open-sesame\",\"protocol\":\"HTTP/2.0\",\"status\":\"300\",\"bytes\":9760,\"referer\":\"https://for.sharp/this/endpoint/prints/money\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:25.843117222Z"}
{"host":"localhost","message":"{\"host\":\"38.66.28.220\",\"user-identifier\":\"KarimMove\",\"datetime\":\"22/Sep/2025:17:25:26\",\"method\":\"PATCH\",\"request\":\"/booper/bopper/mooper/mopper\",\"protocol\":\"HTTP/2.0\",\"status\":\"302\",\"bytes\":39540,\"referer\":\"https://up.mint/secret-info/open-sesame\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:26.842443312Z"}
{"host":"localhost","message":"{\"host\":\"95.224.105.65\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:27\",\"method\":\"PATCH\",\"request\":\"/controller/setup\",\"protocol\":\"HTTP/1.1\",\"status\":\"501\",\"bytes\":36475,\"referer\":\"https://some.coffee/user/booperbot124\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:27.842764515Z"}
{"host":"localhost","message":"{\"host\":\"197.162.35.153\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:28\",\"method\":\"HEAD\",\"request\":\"/secret-info/open-sesame\",\"protocol\":\"HTTP/1.0\",\"status\":\"550\",\"bytes\":34609,\"referer\":\"https://make.itau/this/endpoint/prints/money\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:28.842343264Z"}
{"host":"localhost","message":"{\"host\":\"225.215.17.246\",\"user-identifier\":\"meln1ks\",\"datetime\":\"22/Sep/2025:17:25:29\",\"method\":\"HEAD\",\"request\":\"/do-not-access/needs-work\",\"protocol\":\"HTTP/1.0\",\"status\":\"300\",\"bytes\":11504,\"referer\":\"https://make.visa/observability/metrics/production\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:29.842516190Z"}
{"host":"localhost","message":"{\"host\":\"44.104.173.118\",\"user-identifier\":\"meln1ks\",\"datetime\":\"22/Sep/2025:17:25:30\",\"method\":\"PATCH\",\"request\":\"/do-not-access/needs-work\",\"protocol\":\"HTTP/2.0\",\"status\":\"501\",\"bytes\":29462,\"referer\":\"https://for.casa/user/booperbot124\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:30.842455780Z"}
{"host":"localhost","message":"{\"host\":\"94.238.133.67\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:31\",\"method\":\"POST\",\"request\":\"/secret-info/open-sesame\",\"protocol\":\"HTTP/2.0\",\"status\":\"401\",\"bytes\":49266,\"referer\":\"https://for.careers/this/endpoint/prints/money\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:31.842494758Z"}
{"host":"localhost","message":"{\"host\":\"32.198.110.120\",\"user-identifier\":\"KarimMove\",\"datetime\":\"22/Sep/2025:17:25:32\",\"method\":\"PATCH\",\"request\":\"/apps/deploy\",\"protocol\":\"HTTP/2.0\",\"status\":\"307\",\"bytes\":2102,\"referer\":\"https://for.supplies/controller/setup\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:32.842482780Z"}
{"host":"localhost","message":"{\"host\":\"156.48.43.166\",\"user-identifier\":\"b0rnc0nfused\",\"datetime\":\"22/Sep/2025:17:25:33\",\"method\":\"PATCH\",\"request\":\"/this/endpoint/prints/money\",\"protocol\":\"HTTP/1.1\",\"status\":\"550\",\"bytes\":6737,\"referer\":\"https://up.scholarships/do-not-access/needs-work\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:33.842276138Z"}
{"host":"localhost","message":"{\"host\":\"240.16.218.38\",\"user-identifier\":\"BronzeGamer\",\"datetime\":\"22/Sep/2025:17:25:34\",\"method\":\"PATCH\",\"request\":\"/booper/bopper/mooper/mopper\",\"protocol\":\"HTTP/1.1\",\"status\":\"501\",\"bytes\":24455,\"referer\":\"https://up.ru/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:34.842712690Z"}
{"host":"localhost","message":"{\"host\":\"111.115.2.95\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:35\",\"method\":\"OPTION\",\"request\":\"/booper/bopper/mooper/mopper\",\"protocol\":\"HTTP/2.0\",\"status\":\"501\",\"bytes\":47782,\"referer\":\"https://we.bom/booper/bopper/mooper/mopper\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:35.842652357Z"}
{"host":"localhost","message":"{\"host\":\"225.242.176.18\",\"user-identifier\":\"AmbientTech\",\"datetime\":\"22/Sep/2025:17:25:36\",\"method\":\"GET\",\"request\":\"/booper/bopper/mooper/mopper\",\"protocol\":\"HTTP/2.0\",\"status\":\"400\",\"bytes\":17510,\"referer\":\"https://for.us/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:36.842563338Z"}
{"host":"localhost","message":"{\"host\":\"54.103.95.172\",\"user-identifier\":\"b0rnc0nfused\",\"datetime\":\"22/Sep/2025:17:25:37\",\"method\":\"HEAD\",\"request\":\"/secret-info/open-sesame\",\"protocol\":\"HTTP/1.0\",\"status\":\"403\",\"bytes\":37313,\"referer\":\"https://make.statebank/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:37.843126765Z"}
{"host":"localhost","message":"{\"host\":\"154.115.115.219\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:38\",\"method\":\"POST\",\"request\":\"/do-not-access/needs-work\",\"protocol\":\"HTTP/2.0\",\"status\":\"200\",\"bytes\":33917,\"referer\":\"https://up.sarl/controller/setup\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:38.842404698Z"}
{"host":"localhost","message":"{\"host\":\"5.61.15.83\",\"user-identifier\":\"KarimMove\",\"datetime\":\"22/Sep/2025:17:25:39\",\"method\":\"GET\",\"request\":\"/observability/metrics/production\",\"protocol\":\"HTTP/1.0\",\"status\":\"401\",\"bytes\":7104,\"referer\":\"https://make.mo/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:39.842595810Z"}
Expected behavior
Successful insert
Additional context
Main latest build
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the file.d entry point and reproduce the failure using config.yaml, demo_logs.json, and the Loki output settings shown. Trace how json_decode transforms the message before insertion and verify the behavior against the provided nested JSON logs. Done means the same configuration successfully inserts the records into Loki.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, json
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100