openclaw / openclaw/shellbench

Support exact OpenClaw candidate packages in native evals

Open
#56 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

clawsweeper:linked-pr-open clawsweeper:needs-product-decision clawsweeper:needs-security-review clawsweeper:no-new-fix-pr impact:other impact:security issue-rating: 🌊 off-meta tidepool P2
Dominant language
Python
Stars
139
Forks
30
Avg merge
7m
Merged PRs (30d)
6

Description

Summary

Allow native OpenClaw evaluation campaigns to install an exact local package
tarball instead of requiring a registry-published harness version.

Motivation

ShellBench can reproduce released OpenClaw versions, but it cannot measure a
candidate OpenClaw fix before publication. That blocks clean baseline/candidate
ablations for benchmark-driven product work and encourages ad hoc remote
machine edits that are not auditable or resumable.

The campaign needs to preserve the candidate package name, version, SHA-256,
and resume identity so a result can be tied to the exact bits under test.

Proposed approach

Add an OpenClaw-only --openclaw-package-tarball input that:

  • validates the package name and version before leasing,
  • stages the archive with restrictive permissions,
  • uploads and installs the exact artifact during bootstrap,
  • records its SHA-256 in fleet, run, and audit metadata,
  • rejects resume when candidate identity changes or is missing.

Registry-based runs remain unchanged when the option is absent.

Alternatives considered

  • Publishing temporary npm versions: pollutes the registry and couples
    evaluation to release operations.
  • Editing a warmed machine: not reproducible, not resumable, and not auditable.
  • Building OpenClaw inside every lease: slower and introduces build-environment
    variance into the harness comparison.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the native OpenClaw campaign paths for leasing, bootstrap installation, fleet/run/audit metadata, and resume validation. Map where package inputs are handled and where campaign identity is persisted. Done means an exact local tarball can be validated, installed, hashed, recorded, and rejected on an incompatible resume while registry runs remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.