openclaw / openclaw/shellbench
Support exact OpenClaw candidate packages in native evals
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 139
- Forks
- 30
- Avg merge
- 7m
- Merged PRs (30d)
- 6
Description
Summary
Allow native OpenClaw evaluation campaigns to install an exact local package
tarball instead of requiring a registry-published harness version.
Motivation
ShellBench can reproduce released OpenClaw versions, but it cannot measure a
candidate OpenClaw fix before publication. That blocks clean baseline/candidate
ablations for benchmark-driven product work and encourages ad hoc remote
machine edits that are not auditable or resumable.
The campaign needs to preserve the candidate package name, version, SHA-256,
and resume identity so a result can be tied to the exact bits under test.
Proposed approach
Add an OpenClaw-only --openclaw-package-tarball input that:
- validates the package name and version before leasing,
- stages the archive with restrictive permissions,
- uploads and installs the exact artifact during bootstrap,
- records its SHA-256 in fleet, run, and audit metadata,
- rejects resume when candidate identity changes or is missing.
Registry-based runs remain unchanged when the option is absent.
Alternatives considered
- Publishing temporary npm versions: pollutes the registry and couples
evaluation to release operations. - Editing a warmed machine: not reproducible, not resumable, and not auditable.
- Building OpenClaw inside every lease: slower and introduces build-environment
variance into the harness comparison.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the native OpenClaw campaign paths for leasing, bootstrap installation, fleet/run/audit metadata, and resume validation. Map where package inputs are handled and where campaign identity is persisted. Done means an exact local tarball can be validated, installed, hashed, recorded, and rejected on an incompatible resume while registry runs remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100