microsoft / microsoft/amplifier

[hooks-logging] No retention/rotation/size-cap option for events.jsonl — append-only files reach 1.1 GB per session

Open
#333 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.1k
Forks
261
Avg merge
3h 28m
Merged PRs (30d)
13

Description

Summary

The hooks-logging module is a pure append-only writer — the only file operation is log_path.open("a", ...). There is no size cap, rotation, or TTL option, so events.jsonl grows without bound for the life of a session, and session directories accumulate forever.

Measured impact (one real machine, heavy but ordinary interactive use)

  • 23.8 GB of events.jsonl across 2,892 session files
  • Individual files up to 1.1 GB per session
  • 13.8 GB written in a single 2-day window

(Measured with the app-cli raw: true default active — see #332, which makes full llm:request payloads ~97% of the bytes. But even with raw: false, append-forever with no retention option remains unbounded.)

Why this belongs in the module (mechanism, not policy)

Retention policy is arguably app-layer, and amplifier session cleanup --days N exists. But an opt-in size cap or rotation option on the writer is mechanism, not policy — consistent with kernel philosophy. Today the only mitigation is manual whole-session deletion; there is no way to bound the cost of a single long-running session.

Ask

Optional module config on hooks-logging, e.g.:

hooks:
  - module: hooks-logging
    config:
      max_bytes: 104857600   # opt-in cap or rotation per session log

Related: #332 (default raw: true injection is what makes the growth rate extreme).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the hooks-logging module at the append writer described by log_path.open("a", ...), then trace how module configuration is loaded. Clarify whether max_bytes should cap or rotate each session's events.jsonl, and define behavior for files that reach the limit. Verify the chosen behavior with a long-write case and confirm that normal logging remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.