datalust / datalust/seqcli

`seqcli ingest` extract embedded JSON values from plain text logs

Open
#238 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C#
Stars
167
Forks
30
Avg merge
1d 6h
Merged PRs (30d)
9

Description

Discussed in https://github.com/datalust/seq-tickets/discussions/1536

Originally posted by mkvonarx March 31, 2022
Hi

We're using serilog for our logging. Unfortunately, we cannot user the CLEF JSON format in production. Instead we use a more human readable format with # to delimit the values and a {Properties} at the end to also output all remaining Serilog properties. Our format string looks almost like this:
"outputTemplate": "{Timestamp:yyyy-MM-dd HH:mm:ss.fff} # {ThreadId,3} # {Level:u3} # {SourceContext,-30} # {Message:lj} # {Properties} # {Exception}${NewLine}"

We're trying to ingest these logs to Seq with "seqcli ingest -x" and it works mostly fine. The only thing we cannot manage to nicely ingest is the {Properties} part. This is a JSON inside the human readable text, and seqcli ingest does not have a JSON parser according to https://docs.datalust.co/docs/command-line-client#extraction-patterns

Is there any way to ingest the {Properties} with seqcli ingest as anything else than a string?

Having a 5th builtin property like e.g. @p that directly understands JSON like {Properties} (like @t etc.) would propably be the nicest solution. Or maybe something like "seqcli ingest -x ".... {:json} ..." could also work.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the seqcli ingest -x extraction-pattern behavior and the linked command-line client documentation. The change should let embedded JSON in plain-text log properties be ingested as structured values rather than a string; the payload does not identify implementation files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.