Smart Token Budgeting & Large Diff Truncation for Large Commits

Problem Statement

When working on large commits (such as updating dependency lockfiles (go.sum, package-lock.json), large code refactors, or bulk file additions), the raw git diff can easily exceed LLM context windows or unnecessarily consume a large number of input tokens. Currently, git-commit-summary sends the entire raw diff to the LLM provider.

Proposed Feature: Smart Token Budgeting & Large Diff Truncation

Implement a preprocessing step for staged diffs that ensures large changes fit comfortably within LLM token limits while retaining the essential semantic summary of the changes.

Detailed Outline Plan

1. Diff Statistics & Token Estimation:

   • Parse the git diff output into file-level chunks (internal/git).
   • Implement a lightweight token estimator (e.g., character-to-token heuristic or using a tokenizer helper) to gauge the size of each file diff.

2. Diff Prioritization & Truncation Strategy:

   • Lockfiles & Generated Files: Automatically collapse or omit full diffs for well-known generated files / lockfiles (go.sum, package-lock.json, pnpm-lock.yaml, minified assets), replacing them with a summary line (e.g., ... (lockfile updated: X dependencies added, Y removed)).
   • Code Files: If total diff exceeds a configured threshold (e.g., max token limit or max line count):
     • Prioritize core logic files over documentation or tests, or truncate individual file diffs after a maximum number of hunks/lines, appending a truncation notice (e.g., ... [diff truncated for length]).

3. Configuration & Flags:

   • Add configuration options (in config.env / .git-commit-summary.yaml) for max diff tokens / size thresholds.
   • Add CLI flag support if needed (e.g., --max-diff-tokens).

4. Testing & Validation:

   • Add unit tests in internal/git for diff truncation and lockfile collapsing logic.
   • Verify behavior against mock large diff inputs.

Start by reading the staged-diff handling in internal/git and the configuration surfaces config.env and .git-commit-summary.yaml. Use mock large diff inputs to understand the required internal/git tests. Done means oversized diffs stay within configured limits, lockfiles and generated files are collapsed, and the related configuration or CLI option is covered by tests.