Owner

UKGovernmentBEIS

14 indexed repositories · View on GitHub

inspect_ai
Inspect: A framework for large language model evaluations

Python · 2812 stars
inspect_evals
Collection of evals for Inspect AI

Python · 675 stars
control-arena
ControlArena is a collection of settings, model organisms and protocols - for running control experiments.

Python · 239 stars
vllm-lens
Extract residual-stream activations and apply steering vectors (including activation oracles) to any vLLM model during inference.

Python · 124 stars
inspect_k8s_sandbox
A Kubernetes sandbox environment for use with inspect_ai

Python · 40 stars
optstop
Optimal stopping package for LLM evaluations

Python · 21 stars
inspect_proxmox_sandbox
A Proxmox sandbox environment for use with inspect_ai

Python · 9 stars
inspect_ec2_sandbox
An EC2 sandbox environment for use with inspect_ai

Python · 7 stars
esos
Application code repository for ESOS (Energy Savings Opportunity Scheme)

Java · 3 stars
mets
Source code report for METS (Manage your UK Emissions Trading Scheme reporting service)

Java · 2 stars
ukets-registry
Application code for the UK Emissions Trading Scheme Registry

Java · 2 stars
isambard_containers
Isambard Containers

Python · 1 star
CCA
Climate Change Agreements service application code

Java · 1 star
ukets-maritime

TypeScript · 1 star

Beginner friendly Everything

Open All

bbh: exact-match and Dyck subsets score wrong answers as correct via suffix matching Open

Difficulty 3/5 1-2 days Newbie friendliness 75/100

UKGovernmentBEIS/inspect_evals#2488 · 1 comment · Opened Sep 20, 2026
bbh: declared dataset_samples is 250 but the default task evaluates 6509 Open

Difficulty 1/5 Under an hour Newbie friendliness 92/100

UKGovernmentBEIS/inspect_evals#2487 · 1 comment · Opened Sep 20, 2026
apps: declared dataset_samples is 5000 but the default task evaluates 3000 Open

Difficulty 1/5 Under an hour Newbie friendliness 94/100

UKGovernmentBEIS/inspect_evals#2486 · 1 comment · Opened Sep 20, 2026
Support authenticating via Codex subscription (ChatGPT Plus/Pro) instead of pay-per-token API key Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

UKGovernmentBEIS/inspect_ai#5484 · Opened Sep 20, 2026
Web browser tool directory still documents the pre-split dm_env_rpc design (dead test, stale README output format, diagram, and a masked session error) Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

UKGovernmentBEIS/inspect_ai#5479 · Opened Sep 19, 2026
`--sample-id` glob patterns select the wrong samples on numeric ids, and warn "not found" even when they match Open

Difficulty 3/5 1-2 days Newbie friendliness 55/100

UKGovernmentBEIS/inspect_ai#5478 · Opened Sep 19, 2026
file_dataset() still raises "No input in dataset" on a BOM CSV: its utf-8 default overrides the utf-8-sig that #5361 gave csv_dataset() Open

Difficulty 2/5 1-3 hours Newbie friendliness 58/100

UKGovernmentBEIS/inspect_ai#5474 · Opened Sep 18, 2026
f1() ignores token multiplicity: _to_words returns a set, so a repeated answer scores 1.0 (residual half of #4618) Open

Difficulty 2/5 1-3 hours Newbie friendliness 90/100

UKGovernmentBEIS/inspect_ai#5473 · 2 comments · Opened Sep 18, 2026
Long OpenAI generations time out by default Open

Difficulty 5/5 Over a week Newbie friendliness 48/100

UKGovernmentBEIS/inspect_ai#5471 · Opened Sep 18, 2026
Add native `web_search()` support for OpenRouter models Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

UKGovernmentBEIS/inspect_ai#5468 · Opened Sep 18, 2026
Warn when the output budget is consumed by the reasoning channel (`reasoning_tokens == max_tokens`, empty completion) Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

UKGovernmentBEIS/inspect_ai#5467 · 5 comments · Opened Sep 18, 2026
`mmlu_0_shot` can report 0.000 with no error on a reasoning model — `max_non_cot_tokens` defaults to an API floor (16), not a budget Open

Difficulty 4/5 3-5 days Newbie friendliness 65/100

UKGovernmentBEIS/inspect_evals#2473 · 1 comment · Opened Sep 18, 2026
Adopt the locked sandbox dependency pattern from #2462 across eval Dockerfiles Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 35/100

UKGovernmentBEIS/inspect_evals#2468 · 3 comments · Opened Sep 18, 2026
Register submission for an eval based on non-arXiv published research Open

Difficulty 2/5 1-3 hours Newbie friendliness 55/100

UKGovernmentBEIS/inspect_evals#2467 · Opened Sep 17, 2026
`precomputed_scores()` picks the JSON vs JSON Lines reader from the full URL, so a JSONL scores file behind a query string fails to load Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

UKGovernmentBEIS/inspect_ai#5455 · Opened Sep 17, 2026
TokenInterval checkpoint triggers silently load back as TurnInterval after JSON round-trip Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

UKGovernmentBEIS/inspect_ai#5452 · Opened Sep 17, 2026
SLOP ALERT: move the guest exec wrapper out of ProxmoxSandboxEnvironment Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

UKGovernmentBEIS/inspect_proxmox_sandbox#132 · Opened Sep 16, 2026
exec(timeout=) is ignored when the guest agent is down: QGA retry loop takes ~7 min to fail Open

Difficulty 4/5 3-5 days Newbie friendliness 52/100

UKGovernmentBEIS/inspect_proxmox_sandbox#131 · Opened Sep 16, 2026
[bug] AgentThreatBench: agents report on prompt injection attempts, get marked down Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 78/100

UKGovernmentBEIS/inspect_evals#2460 · Opened Sep 16, 2026
Evaluator-grade eval logs: per-sample records, external registration, selective disclosure Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

UKGovernmentBEIS/inspect_ai#5444 · Opened Sep 16, 2026
math() cannot distinguish an empty completion from malformed output, and misses most real format violations Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

UKGovernmentBEIS/inspect_ai#5441 · 4 comments · Opened Sep 16, 2026
[Bug] sycophancy: multiple-choice grader scores CORRECT when the target letter appears anywhere in the reply Open

Difficulty 3/5 1-2 days Newbie friendliness 58/100

UKGovernmentBEIS/inspect_evals#2457 · 1 comment · Opened Sep 16, 2026
Adopt shared fenced-code extraction across evaluations Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 52/100

UKGovernmentBEIS/inspect_evals#2454 · 1 comment · Opened Sep 16, 2026
PaperBench: record the reproduce.sh exit status in reproduction metadata Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 74/100

UKGovernmentBEIS/inspect_evals#2453 · 1 comment · Opened Sep 16, 2026
PaperBench: verified Kubernetes sandbox support Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 45/100

UKGovernmentBEIS/inspect_evals#2451 · Opened Sep 16, 2026
PaperBench: resolve judge and parser models through model roles, defaulting the parser to gpt-4o-2024-08-06 Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 68/100

UKGovernmentBEIS/inspect_evals#2450 · Opened Sep 16, 2026
[Bug] agentdojo reads suite YAML with the locale encoding, so travel user tasks 9 and 11 score I under a non-UTF-8 locale Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

UKGovernmentBEIS/inspect_evals#2449 · Opened Sep 16, 2026
Windows: a sample error whose message has a character outside the ANSI code page aborts the task and hides the real error Open

Difficulty 3/5 1-2 days Newbie friendliness 74/100

UKGovernmentBEIS/inspect_ai#5434 · Opened Sep 16, 2026
Every model call fails on Windows with a CJK system code page: bundled model info YAML is read with the locale encoding Open

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

UKGovernmentBEIS/inspect_ai#5433 · 3 comments · Opened Sep 16, 2026
[Bug]: SAE side-task scorer turns an unparseable judge result into success Open

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

UKGovernmentBEIS/control-arena#878 · Opened Sep 16, 2026
math(): an over-limit or code-shaped answer candidate raises out of the scorer instead of scoring INCORRECT Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

UKGovernmentBEIS/inspect_ai#5430 · 1 comment · Opened Sep 15, 2026
Proposal: parallel approval and review chains (a dict of named policy lists) Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

UKGovernmentBEIS/inspect_ai#5423 · 2 comments · Opened Sep 15, 2026
`computer` tool: `back_click` / `forward_click` are offered by the host but rejected by the container, and the failure ends the sample Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

UKGovernmentBEIS/inspect_ai#5421 · 1 comment · Opened Sep 15, 2026
A model-supplied date/time tool argument crashes the sample instead of coming back as a retryable error Open

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

UKGovernmentBEIS/inspect_ai#5415 · Opened Sep 15, 2026
A tool argument the model fills wrongly ends the sample instead of coming back as a retryable error Open

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

UKGovernmentBEIS/inspect_ai#5403 · Opened Sep 15, 2026
Inspect View: enforce per-panel scope in the view server (scoped authorization tokens) Open

accepted enhancement viewer

Difficulty 5/5 Over a week Newbie friendliness 28/100

UKGovernmentBEIS/inspect_ai#5400 · Opened Sep 14, 2026
[Bug] AgentHarm benign file tools fail outside the package working directory Open

Difficulty 3/5 1-2 days Newbie friendliness 78/100

UKGovernmentBEIS/inspect_evals#2439 · Opened Sep 14, 2026
`computer` tool: `coordinate` description names 2 actions; 9 require it and 2 more act on it Open

accepted bug good first issue

Difficulty 3/5 1-2 days Newbie friendliness 58/100

UKGovernmentBEIS/inspect_ai#5386 · 3 comments · Opened Sep 14, 2026
Map Compose no-new-privileges to allowPrivilegeEscalation Open

Difficulty 3/5 1-2 days Newbie friendliness 72/100

UKGovernmentBEIS/inspect_k8s_sandbox#258 · 2 comments · Opened Sep 14, 2026
bash_session sends the literal string `None` to the shell when `input` is omitted for `type_submit` Open

accepted bug good first issue

Difficulty 1/5 1-3 hours Newbie friendliness 88/100

UKGovernmentBEIS/inspect_ai#5379 · 4 comments · Opened Sep 14, 2026
Sorted-input warning misses natural ID order and never reaches the CLI console Open

Difficulty 3/5 1-2 days Newbie friendliness 72/100

UKGovernmentBEIS/optstop#8 · Opened Sep 13, 2026
match(), includes(), exact() and f1() never set Score.reason, so an empty completion and a refusal are indistinguishable from a wrong answer Open

accepted bug

Difficulty 3/5 1-2 days Newbie friendliness 76/100

UKGovernmentBEIS/inspect_ai#5376 · 5 comments · Opened Sep 13, 2026
JSON dataset loading silently drops sample checkpoint settings Open

accepted bug

Difficulty 3/5 1-2 days Newbie friendliness 78/100

UKGovernmentBEIS/inspect_ai#5374 · 2 comments · Opened Sep 13, 2026
Lowercase `z` UTC ISO timestamps fail on Python 3.11+ despite the normalizer accepting `z` Open

accepted bug good first issue

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

UKGovernmentBEIS/inspect_ai#5369 · 3 comments · Opened Sep 12, 2026
`--env` mangles comma-containing environment variable values into Python-list strings Open

accepted bug

Difficulty 2/5 1-3 hours Newbie friendliness 88/100

UKGovernmentBEIS/inspect_ai#5368 · 4 comments · Opened Sep 12, 2026
Proposal: an embodied-policy eval with a matched benign control arm Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

UKGovernmentBEIS/inspect_evals#2423 · Opened Sep 12, 2026
Agent bridges under-report cached prompt tokens in Responses and Gemini usage Open

Difficulty 4/5 3-5 days Newbie friendliness 70/100

UKGovernmentBEIS/inspect_ai#5364 · 2 comments · Opened Sep 12, 2026
Gemini: GOOGLE_USE_ADC overrides an explicitly supplied API key Open

Difficulty 4/5 3-5 days Newbie friendliness 68/100

UKGovernmentBEIS/inspect_ai#5358 · 1 comment · Opened Sep 11, 2026
Proposal (not yet specified): MetricResult so a metric can report its own denominator and why it is undefined Open

UKGovernmentBEIS/inspect_ai#5352 · 1 assignee · Opened Sep 11, 2026
Remove redundant epoch-count task parameters and document runner overrides Open

documentation enhancement

Difficulty 4/5 3-5 days Newbie friendliness 55/100

UKGovernmentBEIS/inspect_evals#2419 · 1 comment · Opened Sep 11, 2026

Showing the newest 100

This page lists what was indexed most recently. Advanced filter has the whole inventory, narrowed by language, difficulty and how long a task takes.

Open advanced filter

UKGovernmentBEIS

Showing the newest 100

Get new issues in your inbox