mpfaffenberger / mpfaffenberger/code_puppy
Add first-class Harbor / Terminal-Bench 2.0 support for leaderboard evaluation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 814
- Forks
- 278
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 76
Description
Summary
Please add first-class Harbor support so Code Puppy can run reproducibly on Terminal-Bench 2.0 and produce a valid submission for the official leaderboard.
Harbor is the official Terminal-Bench 2.0 harness. Code Puppy currently has a direct CLI, but it is not available as a supported Harbor agent, which makes full benchmark runs and apples-to-apples leaderboard comparison difficult.
Requested support
- Add or upstream a Harbor adapter for Code Puppy, preferably a
BaseInstalledAgentimplementation that can be selected with--agent/-a. - Provide a stable, fully non-interactive/headless invocation that accepts the task instruction, runs from Harbor's task working directory, and returns an accurate exit status.
- Install and run correctly in fresh Harbor task containers without depending on host-global Code Puppy state. Pin or record the Code Puppy version used for each run.
- Pass only the selected provider/model credentials into the agent process; do not persist secrets in Harbor job configuration or logs.
- Preserve useful run artifacts (agent output/trajectory, model identity, errors, timing, and token usage where available) so trials can be audited.
- Support the architecture/environment needed for official leaderboard runs (at minimum native Linux amd64), with clear errors for unsupported environments.
- Document a reproducible smoke test and full benchmark command, for example:
harbor run \
-d terminal-bench/terminal-bench-2 \
-a code-puppy \
-m <provider/model>
- Document the process for packaging the completed Harbor job and submitting it to the Terminal-Bench 2 leaderboard repository.
Acceptance criteria
- A clean Harbor install can discover or load the Code Puppy agent adapter.
- One pinned Terminal-Bench 2.0 task completes end-to-end and produces a verifier reward plus inspectable agent artifacts.
- A full
terminal-bench/terminal-bench-2run can complete without interactive prompts. - Failed agent runs are distinguishable from environment/install/verifier failures.
- The generated job metadata contains the Code Puppy version and model configuration, but no credentials.
- The resulting job is in the format required for an official leaderboard submission.
This would make Code Puppy directly comparable with the other coding agents already integrated into Harbor and enable it to be ranked on the official Terminal-Bench leaderboard.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository files or tests are named. Start by locating the existing direct CLI and the Harbor BaseInstalledAgent integration point, then run the one-task Harbor smoke test from a clean environment. Done means a pinned task and full run work headlessly, produce inspectable artifacts and metadata without credentials, and distinguish agent, environment, install, and verifier failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100