microsoft / microsoft/STATE-Bench
Question: sanctioned validation path for custom ContextContract agent
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 91
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
Request: sanctioned validation path for ContextContract agent
I am testing a local ContextContract controller with STATE-Bench.
Current status:
- A custom
BaseAgentloads in the public checkout. - A direct no-score
run_task()smoke works on two shopping tasks. - I am not claiming an official score or external validation.
- The official CLI path is blocked locally by
STATE_BENCH_EVAL_ENDPOINT.
Question:
What is the clean sanctioned path from adapter compatibility to valid STATE-Bench evidence?
Useful answers would be:
- how to access/configure the locked eval endpoint,
- whether a deterministic no-score smoke can count as nonofficial adapter evidence,
- whether this belongs in Main Track or Agent Learning Track.
I want to avoid private simulator hacks or protocol-unsafe claims.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the public BaseAgent adapter and the official CLI path, then trace where STATE_BENCH_EVAL_ENDPOINT is required. Review the existing no-score run_task smoke and the benchmark track definitions. Done means documenting the sanctioned endpoint configuration, the status of nonofficial smoke evidence, and the appropriate track.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, testing-qa
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100