HarperFast / HarperFast/harper

Structured log emission: JSON format option, multi-line-safe entries, ANSI stripping

Open
#1,637 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Problem

The 2026-07-06 Datadog audit showed the log *format* itself is a major cost and signal problem:

- **~87% of "error"-status events in Datadog aren't errors.** Harper writes all output to stderr, and the Docker json-file → Datadog agent path tags stderr lines as `error`. Of ~298M "error" events in 7 days, ~94M were `[warn]`-level messages and ~164M were stack-trace/object-dump fragments; only ~40M contained `[error]`.
- **Multi-line entries are billed per line.** Stack traces and object dumps (e.g. the MQTT auth-failure cert dump) emit each line as a separate log event — including lines that are just `}` or ` at X (...)`. This destroys pattern clustering and multiplies billed events.
- **ANSI color escapes ship verbatim** (`[32m...[39m`), inflating bytes and breaking message clustering.

## Proposal

1. `logging.format: json` config option: one JSON object per line — `{time, level, thread, tags/component, audience, msg, err?}` with the error stack as a single embedded string field. One event per entry by construction; the agent maps `level` correctly; clustering works.
2. When output is not a TTY (or when format is json), strip/disable ANSI codes.
3. Keep current plaintext as the default for local/dev; Fabric flips to json via config.
4. Route by level: only actual error/fatal to stderr (or keep everything on stdout with the level in the payload — with json format the stream no longer needs to carry severity).

Interim agent-side mitigation (multi-line aggregation rules keyed on the ISO-timestamp prefix, status remapping) is filed in host-manager; this issue is the at-source fix.

## Related
harper-pro#262 (dedup/run-counts), PR #372 (status bridge), host-manager agent-pipeline issue.

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating Harper's logging configuration and emission path, then review the related host-manager mitigation and Fabric configuration; done means JSON entries are single-line and structured, ANSI is removed when appropriate, plaintext remains the default, and severity routing is defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.