GaiaLogic Submission: Agent Learning Track on nano

Hello,

Thank you for the review and questions. Since we didn't hear back on which of the
two variants to submit, we are proceeding with our stronger result — a
label-free, test-adapted system — with full disclosure below. If you would
rather have the never-test-informed variant (gaia-nano-v3) for the Agent
Learning Track, we can swap to it immediately; it is ready.

System: gaia-nano — the entire agentic system runs on gpt-5.4-nano

gaia-nano is an offline, deterministic, label-free rule-learning system for the
Agent Learning Track. Its defining property is that the whole agentic system
runs on a single small model, gpt-5.4-nano, and nothing larger anywhere:

• Rule learning is fully deterministic — no LLM call at all. Per-domain
  if/else rules are mined offline from observable regularities in the train task
  trajectories (conduct patterns, tool paths, and policy branches surfaced from
  the environment documents the agent observed). There is no model in the learning
  stage.
• The only model in the loop is the gpt-5.4-nano agent under test
  (reasoning high), served the mined rules through a read-only
  retrieve_learnings hook.
• We use no gpt-5.4 (full), no separate planner/reasoning model, and no LLM
  for rule induction. The gpt-5.4 simulator and judge are the locked
  STATE-Bench harness, not part of our system.

(For reproducibility against the frozen scored trajectories, the internal code and
artifact names remain compliant_forge / CompliantForgeAgent.)

Submission summary

• Track: Agent Learning Track
• System: gaia-nano (offline, deterministic, label-free rule learning; nano-only)
• Benchmark version: 0.8.0
• Evaluation protocol: state_bench_v0.8.0_gpt54 (locked Azure GPT-5.4
  simulator + judge, judge reasoning high)
• Agent under test: CompliantForgeAgent — OpenAI gpt-5.4-nano, reasoning high
• Coverage: 50 held-out test tasks × 5 runs per domain (750 scored
  trajectories total)

Results (50 test tasks × 5 runs per domain)

Compliance disclosure (please read first)

To be fully transparent, and consistent with our earlier reply in this thread:
this submission is label-free but test-adapted at the design level.

• No labels, task descriptions, or any other task/environment field — from
  either the train or the test set — were ever fed into the agent or the learning
  harness. The only learning input is
  datasets/train_task_trajectories/<domain>/ (the recorded conversation + tool
  calls the protocol-locked agent observed). This is enforced at load time by an
  input boundary (allowed_inputs.py) that refuses any label/oracle/judge field
  and any path outside train_task_trajectories/.
• However, an earlier iteration of this system was run on the test split and
  our engineers examined those results. Based on that examination we made general
  architectural changes — primarily prompt engineering / conduct-rail design —
  which yielded a moderate improvement. This was a one-time, human-in-the-loop
  design adaptation. The submitted frozen artifact was not itself changed after
  its scored run, but its design was informed by inspecting prior test-set
  behavior. We disclose this explicitly rather than leave it implicit.
• We also have gaia-nano-v3, which is slightly weaker but was never informed
  by the test set in any way (derived solely from train trajectories), in case
  you prefer it.

Attached documents (what each one is)

• outputs.zip — the required submission artifact. Layout
  outputs/<domain>/run{1..5}/<task_id>.json + outputs/<domain>/metrics.json
  for all three domains. Contains all 750 judge-scored trajectories; each
  metrics.json carries the protocol id state_bench_v0.8.0_gpt54 and the public
  metrics reported above.
• SUBMISSION.md — the method write-up: the nano-only design, the
  compliance-by-construction input boundary, the deterministic (no-LLM) rule
  mining into per-family if/else rules, the read-only retrieve_learnings
  retrieval hook, agent wiring, and the exact evaluation configuration. Includes
  the test-set adaptation disclosure above.
• contamination_proof.md / contamination_proof.json — machine-checked
  proof that the learned rules were built only from train trajectories and that
  the set of train ids cited as rule evidence has zero overlap with the test
  split (covered ∩ test = 0, outside-split = 0) for all three domains, with
  ground_truth_used = false / labels_used = false re-affirmed from artifact
  provenance. Reproducible via the bundled contamination_check.py.
• harness_integrity.md — proof that the locked STATE-Bench surface is
  unmodified (empty git diff on tracked files), both repository commit SHAs, and
  the protocol pin. Only the documented agents/ user-extension is added.
• final_run_manifest.md — full run registration: exact run_batch and
  compute_metrics commands per domain, the frozen configuration
  (num_runs=5, top_k=3, workers=4), the mined-artifact fingerprint per domain, and
  the file-accurate results table.

outputs.zip

SUBMISSION.md
harness_integrity.md
final_run_manifest.md
contamination_proof.md
contamination_proof.json
contamination_check.py

Best wishes,
Simon

Start by reading SUBMISSION.md, final_run_manifest.md, and contamination_proof.md, then run or inspect contamination_check.py. Compare the disclosed test-set adaptation and the two available variants against the Agent Learning Track requirements. Done means deciding whether gaia-nano or gaia-nano-v3 is acceptable and recording the review outcome.