mpfaffenberger / mpfaffenberger/code_puppy

Add first-class Harbor / Terminal-Bench 2.0 support for leaderboard evaluation

Open
#667 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
814
Forks
278
Avg merge
2d 5h
Merged PRs (30d)
76

Description

Summary

Please add first-class Harbor support so Code Puppy can run reproducibly on Terminal-Bench 2.0 and produce a valid submission for the official leaderboard.

Harbor is the official Terminal-Bench 2.0 harness. Code Puppy currently has a direct CLI, but it is not available as a supported Harbor agent, which makes full benchmark runs and apples-to-apples leaderboard comparison difficult.

Requested support

  • Add or upstream a Harbor adapter for Code Puppy, preferably a BaseInstalledAgent implementation that can be selected with --agent / -a.
  • Provide a stable, fully non-interactive/headless invocation that accepts the task instruction, runs from Harbor's task working directory, and returns an accurate exit status.
  • Install and run correctly in fresh Harbor task containers without depending on host-global Code Puppy state. Pin or record the Code Puppy version used for each run.
  • Pass only the selected provider/model credentials into the agent process; do not persist secrets in Harbor job configuration or logs.
  • Preserve useful run artifacts (agent output/trajectory, model identity, errors, timing, and token usage where available) so trials can be audited.
  • Support the architecture/environment needed for official leaderboard runs (at minimum native Linux amd64), with clear errors for unsupported environments.
  • Document a reproducible smoke test and full benchmark command, for example:
harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a code-puppy \
  -m <provider/model>

Acceptance criteria

  • A clean Harbor install can discover or load the Code Puppy agent adapter.
  • One pinned Terminal-Bench 2.0 task completes end-to-end and produces a verifier reward plus inspectable agent artifacts.
  • A full terminal-bench/terminal-bench-2 run can complete without interactive prompts.
  • Failed agent runs are distinguishable from environment/install/verifier failures.
  • The generated job metadata contains the Code Puppy version and model configuration, but no credentials.
  • The resulting job is in the format required for an official leaderboard submission.

This would make Code Puppy directly comparable with the other coding agents already integrated into Harbor and enable it to be ranked on the official Terminal-Bench leaderboard.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository files or tests are named. Start by locating the existing direct CLI and the Harbor BaseInstalledAgent integration point, then run the one-task Harbor smoke test from a clean environment. Done means a pinned task and full run work headlessly, produce inspectable artifacts and metadata without credentials, and distinguish agent, environment, install, and verifier failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.