ozontech / ozontech/file.d

Bug: failed loki insert

Open
#899 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
504
Forks
263
Avg merge
4d 2h
Merged PRs (30d)
5

Description

Describe the bug
if log line is json the insertion would fail

{"level":"error","ts":"2025-09-28T12:21:44.721518Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:38660}]]}]}|..., bigger context ...|tocol\":\"HTTP/2.0\",\"status\":\"304\",\"bytes\":38660}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.722507Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:17003}]]}]}|..., bigger context ...|tocol\":\"HTTP/2.0\",\"status\":\"503\",\"bytes\":17003}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.72251Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|\":2521}]]}]}|..., bigger context ...|otocol\":\"HTTP/1.0\",\"status\":\"410\",\"bytes\":2521}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.72286Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:26831}]]}]}|..., bigger context ...|tocol\":\"HTTP/1.1\",\"status\":\"200\",\"bytes\":26831}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.725175Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:42978}]]}]}|..., bigger context ...|tocol\":\"HTTP/1.0\",\"status\":\"404\",\"bytes\":42978}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.725769Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:41531}]]}]}|..., bigger context ...|tocol\":\"HTTP/1.0\",\"status\":\"301\",\"bytes\":41531}]]}]}|...\n"}
{"level":"error","ts":"2025-09-28T12:21:44.985293Z","logger":"fd.loki.output.loki","message":"can't send data to Loki","address":"http://localhost:3100","error":"bad response: code=400, err=response status from http://localhost:3100/loki/api/v1/push isn't OK: status=400, body=loghttp.PushRequest.Streams: []loghttp.LogProtoStream: unmarshalerDecoder: Value is string, but can't find closing '\"' symbol, error found in #10 byte of ...|:22297}]]}]}|..., bigger context ...|\"HTTP/1.1\",\"status\":\"300\",\"bytes\":22297}]]}]}|...\n"}

To Reproduce
Steps to reproduce the behavior:

  1. file.d --config=./config.yaml
  2. config.yaml
pipelines:
  loki:
    settings:
      capacity: 1000
    input:
      type: file
      persistence_mode: async
      # offsets_op: reset
      paths:
        include:
          - '/Users/romanchechyotkin/file.d/demo_logs.json'
      offsets_file: ./offsets.yaml
    actions:
      - type: json_decode
        field: message
      - type: set_time
        field: timestamp
        override: false
      - type: convert_date
        field: timestamp
        target_format: unixtimenano
        source_formats:
        - 'rfc3339'
        - 'rfc3339nano'
        - '02/Jan/2006:15:04:05 -0700'
        - '2006-01-02T15:04:05.999Z0700'
    output:
      type: loki
      fatal_on_failed_insert: true
      retry: 3
      address: http://localhost:3100
      timestamp_field: timestamp
      message_field: message
  1. demo_logs.json
{"host":"localhost","message":"{\"host\":\"143.61.119.90\",\"user-identifier\":\"jesseddy\",\"datetime\":\"22/Sep/2025:17:25:23\",\"method\":\"HEAD\",\"request\":\"/user/booperbot124\",\"protocol\":\"HTTP/1.0\",\"status\":\"404\",\"bytes\":14476,\"referer\":\"https://we.bauhaus/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:23.842787109Z"}
{"host":"localhost","message":"{\"host\":\"53.108.32.5\",\"user-identifier\":\"KarimMove\",\"datetime\":\"22/Sep/2025:17:25:24\",\"method\":\"DELETE\",\"request\":\"/wp-admin\",\"protocol\":\"HTTP/1.1\",\"status\":\"503\",\"bytes\":28124,\"referer\":\"https://random.xn--ses554g/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:24.842790741Z"}
{"host":"localhost","message":"{\"host\":\"84.243.110.238\",\"user-identifier\":\"jesseddy\",\"datetime\":\"22/Sep/2025:17:25:25\",\"method\":\"PATCH\",\"request\":\"/secret-info/open-sesame\",\"protocol\":\"HTTP/2.0\",\"status\":\"300\",\"bytes\":9760,\"referer\":\"https://for.sharp/this/endpoint/prints/money\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:25.843117222Z"}
{"host":"localhost","message":"{\"host\":\"38.66.28.220\",\"user-identifier\":\"KarimMove\",\"datetime\":\"22/Sep/2025:17:25:26\",\"method\":\"PATCH\",\"request\":\"/booper/bopper/mooper/mopper\",\"protocol\":\"HTTP/2.0\",\"status\":\"302\",\"bytes\":39540,\"referer\":\"https://up.mint/secret-info/open-sesame\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:26.842443312Z"}
{"host":"localhost","message":"{\"host\":\"95.224.105.65\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:27\",\"method\":\"PATCH\",\"request\":\"/controller/setup\",\"protocol\":\"HTTP/1.1\",\"status\":\"501\",\"bytes\":36475,\"referer\":\"https://some.coffee/user/booperbot124\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:27.842764515Z"}
{"host":"localhost","message":"{\"host\":\"197.162.35.153\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:28\",\"method\":\"HEAD\",\"request\":\"/secret-info/open-sesame\",\"protocol\":\"HTTP/1.0\",\"status\":\"550\",\"bytes\":34609,\"referer\":\"https://make.itau/this/endpoint/prints/money\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:28.842343264Z"}
{"host":"localhost","message":"{\"host\":\"225.215.17.246\",\"user-identifier\":\"meln1ks\",\"datetime\":\"22/Sep/2025:17:25:29\",\"method\":\"HEAD\",\"request\":\"/do-not-access/needs-work\",\"protocol\":\"HTTP/1.0\",\"status\":\"300\",\"bytes\":11504,\"referer\":\"https://make.visa/observability/metrics/production\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:29.842516190Z"}
{"host":"localhost","message":"{\"host\":\"44.104.173.118\",\"user-identifier\":\"meln1ks\",\"datetime\":\"22/Sep/2025:17:25:30\",\"method\":\"PATCH\",\"request\":\"/do-not-access/needs-work\",\"protocol\":\"HTTP/2.0\",\"status\":\"501\",\"bytes\":29462,\"referer\":\"https://for.casa/user/booperbot124\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:30.842455780Z"}
{"host":"localhost","message":"{\"host\":\"94.238.133.67\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:31\",\"method\":\"POST\",\"request\":\"/secret-info/open-sesame\",\"protocol\":\"HTTP/2.0\",\"status\":\"401\",\"bytes\":49266,\"referer\":\"https://for.careers/this/endpoint/prints/money\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:31.842494758Z"}
{"host":"localhost","message":"{\"host\":\"32.198.110.120\",\"user-identifier\":\"KarimMove\",\"datetime\":\"22/Sep/2025:17:25:32\",\"method\":\"PATCH\",\"request\":\"/apps/deploy\",\"protocol\":\"HTTP/2.0\",\"status\":\"307\",\"bytes\":2102,\"referer\":\"https://for.supplies/controller/setup\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:32.842482780Z"}
{"host":"localhost","message":"{\"host\":\"156.48.43.166\",\"user-identifier\":\"b0rnc0nfused\",\"datetime\":\"22/Sep/2025:17:25:33\",\"method\":\"PATCH\",\"request\":\"/this/endpoint/prints/money\",\"protocol\":\"HTTP/1.1\",\"status\":\"550\",\"bytes\":6737,\"referer\":\"https://up.scholarships/do-not-access/needs-work\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:33.842276138Z"}
{"host":"localhost","message":"{\"host\":\"240.16.218.38\",\"user-identifier\":\"BronzeGamer\",\"datetime\":\"22/Sep/2025:17:25:34\",\"method\":\"PATCH\",\"request\":\"/booper/bopper/mooper/mopper\",\"protocol\":\"HTTP/1.1\",\"status\":\"501\",\"bytes\":24455,\"referer\":\"https://up.ru/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:34.842712690Z"}
{"host":"localhost","message":"{\"host\":\"111.115.2.95\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:35\",\"method\":\"OPTION\",\"request\":\"/booper/bopper/mooper/mopper\",\"protocol\":\"HTTP/2.0\",\"status\":\"501\",\"bytes\":47782,\"referer\":\"https://we.bom/booper/bopper/mooper/mopper\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:35.842652357Z"}
{"host":"localhost","message":"{\"host\":\"225.242.176.18\",\"user-identifier\":\"AmbientTech\",\"datetime\":\"22/Sep/2025:17:25:36\",\"method\":\"GET\",\"request\":\"/booper/bopper/mooper/mopper\",\"protocol\":\"HTTP/2.0\",\"status\":\"400\",\"bytes\":17510,\"referer\":\"https://for.us/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:36.842563338Z"}
{"host":"localhost","message":"{\"host\":\"54.103.95.172\",\"user-identifier\":\"b0rnc0nfused\",\"datetime\":\"22/Sep/2025:17:25:37\",\"method\":\"HEAD\",\"request\":\"/secret-info/open-sesame\",\"protocol\":\"HTTP/1.0\",\"status\":\"403\",\"bytes\":37313,\"referer\":\"https://make.statebank/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:37.843126765Z"}
{"host":"localhost","message":"{\"host\":\"154.115.115.219\",\"user-identifier\":\"AnthraX\",\"datetime\":\"22/Sep/2025:17:25:38\",\"method\":\"POST\",\"request\":\"/do-not-access/needs-work\",\"protocol\":\"HTTP/2.0\",\"status\":\"200\",\"bytes\":33917,\"referer\":\"https://up.sarl/controller/setup\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:38.842404698Z"}
{"host":"localhost","message":"{\"host\":\"5.61.15.83\",\"user-identifier\":\"KarimMove\",\"datetime\":\"22/Sep/2025:17:25:39\",\"method\":\"GET\",\"request\":\"/observability/metrics/production\",\"protocol\":\"HTTP/1.0\",\"status\":\"401\",\"bytes\":7104,\"referer\":\"https://make.mo/wp-admin\"}","service":"vector","source_type":"demo_logs","timestamp":"2025-09-22T10:25:39.842595810Z"}

Expected behavior
Successful insert

Additional context
Main latest build

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the file.d entry point and reproduce the failure using config.yaml, demo_logs.json, and the Loki output settings shown. Trace how json_decode transforms the message before insertion and verify the behavior against the provided nested JSON logs. Done means the same configuration successfully inserts the records into Loki.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, json
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.