microsoft / microsoft/STATE-Bench

Question: sanctioned validation path for custom ContextContract agent

Open
#43 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
91
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Request: sanctioned validation path for ContextContract agent

I am testing a local ContextContract controller with STATE-Bench.

Current status:

  • A custom BaseAgent loads in the public checkout.
  • A direct no-score run_task() smoke works on two shopping tasks.
  • I am not claiming an official score or external validation.
  • The official CLI path is blocked locally by STATE_BENCH_EVAL_ENDPOINT.

Question:

What is the clean sanctioned path from adapter compatibility to valid STATE-Bench evidence?

Useful answers would be:

  1. how to access/configure the locked eval endpoint,
  2. whether a deterministic no-score smoke can count as nonofficial adapter evidence,
  3. whether this belongs in Main Track or Agent Learning Track.

I want to avoid private simulator hacks or protocol-unsafe claims.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the public BaseAgent adapter and the official CLI path, then trace where STATE_BENCH_EVAL_ENDPOINT is required. Review the existing no-score run_task smoke and the benchmark track definitions. Done means documenting the sanctioned endpoint configuration, the status of nonofficial smoke evidence, and the appropriate track.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, testing-qa
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.