HarperFast / HarperFast/harper
Structured log emission: JSON format option, multi-line-safe entries, ANSI stripping
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Problem
The 2026-07-06 Datadog audit showed the log *format* itself is a major cost and signal problem:
- **~87% of "error"-status events in Datadog aren't errors.** Harper writes all output to stderr, and the Docker json-file → Datadog agent path tags stderr lines as `error`. Of ~298M "error" events in 7 days, ~94M were `[warn]`-level messages and ~164M were stack-trace/object-dump fragments; only ~40M contained `[error]`.
- **Multi-line entries are billed per line.** Stack traces and object dumps (e.g. the MQTT auth-failure cert dump) emit each line as a separate log event — including lines that are just `}` or ` at X (...)`. This destroys pattern clustering and multiplies billed events.
- **ANSI color escapes ship verbatim** (`[32m...[39m`), inflating bytes and breaking message clustering.
## Proposal
1. `logging.format: json` config option: one JSON object per line — `{time, level, thread, tags/component, audience, msg, err?}` with the error stack as a single embedded string field. One event per entry by construction; the agent maps `level` correctly; clustering works.
2. When output is not a TTY (or when format is json), strip/disable ANSI codes.
3. Keep current plaintext as the default for local/dev; Fabric flips to json via config.
4. Route by level: only actual error/fatal to stderr (or keep everything on stdout with the level in the payload — with json format the stream no longer needs to carry severity).
Interim agent-side mitigation (multi-line aggregation rules keyed on the ISO-timestamp prefix, status remapping) is filed in host-manager; this issue is the at-source fix.
## Related
harper-pro#262 (dedup/run-counts), PR #372 (status bridge), host-manager agent-pipeline issue.
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating Harper's logging configuration and emission path, then review the related host-manager mitigation and Fabric configuration; done means JSON entries are single-line and structured, ANSI is removed when appropriate, plaintext remains the default, and severity routing is defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100