Integrate a SPACE benchmark task into the agent-eval framework

Integrate a SPACE benchmark task into the agent-eval framework

Context

DimOS PR #3378 introduces the initial agent-evaluation framework and a direct frozen-memory QA case. This follow-up should use that framework to evaluate the DimOS agent on an existing external benchmark.

SPACE (Spatial Perception And Cognition Evaluation) tests spatial reasoning through text, images, and interactive tasks. The benchmark includes both large-scale environmental reasoning and small-scale object/layout reasoning.

Objective

Integrate one representative SPACE task into the framework from PR #3378 and run the DimOS agent on a bounded, reproducible subset of that task.

Analyze the benchmark, select an appropriate first task and presentation, and explain the selection.

Scope

• Build on the contracts and execution path introduced by PR #3378; do not create a separate evaluation system.
• Select one SPACE task that can be integrated and run end to end within a bounded effort.
• Preserve the benchmark's question, input, answer, and scoring semantics.
• Use SPACE's official evaluation and scoring implementation as-is, without modifying or reimplementing it. DimOS may adapt execution inputs and outputs only at the integration boundary.
• Provide a direct command that another developer can run and test.
• Run a fixed subset and summarize accuracy, invalid responses, and infrastructure failures.
• Produce an evaluation report that analyzes the agent's performance on the selected task and proposes evidence-based improvement directions.

Deliverables

1. Benchmark/task selection note with the reasoning and known limitations.
2. Working integration using the PR #3378 framework, a PR that stacking on it.
3. Reproducible setup and run instructions.
4. An evaluation report covering results, observed strengths and failure modes, and proposed improvement directions.
5. Focused automated tests.

Acceptance criteria

• A fresh checkout can acquire the required upstream data and run the documented command without manual data rewriting.
• The run produces per-case results and an aggregate score traceable to the selected SPACE examples.
• Final scores come directly from the unmodified official SPACE evaluator at the recorded upstream revision.
• The agent uses the DimOS evaluation execution path, not SPACE's built-in model wrappers.
• Provide evaluation report (as PR description) stating the model/configuration, subset selection, and official score; analyzes observed strengths and failure modes; and proposes concrete improvement directions tied to those observations.

Data and licensing constraint

SPACE code uses the Apple Sample Code License, and its dataset uses CC BY-NC-ND 4.0. The implementation must not vendor, modify, or redistribute benchmark data without confirming that the intended use complies with those terms. Prefer an explicit external download/setup step.

References

• Paper: Does Spatial Cognition Emerge in Frontier Models?
• Official SPACE repository
• SPACE dataset license
• Prerequisite DimOS evaluation framework PR #3378

Start by reading the contracts and execution path introduced by PR #3378, then inspect the official SPACE repository to select one bounded task and its required external setup. Define the integration boundary, focused automated tests, reproducible run command, and report; done means per-case results and an aggregate score from the unmodified official evaluator without vendored benchmark data.