[bird_sql] Add versioned, configurable SQL result-comparison modes

Use cases, pain points, and background

Execution-based SQL benchmarks do not all share the same result semantics. resources_servers/bird_sql currently compares set(gold_rows) == set(pred_rows), matching its established BIRD behavior but discarding duplicate-row multiplicity. SQL normally has bag semantics, and duplicate multiplicity can be part of the answer. Other tasks may also require order-sensitive comparison.

Changing BIRD's established default would break historical comparability. The generalized need is therefore an explicit, versioned comparison contract with backward-compatible modes—not a new default tied to any model or grounding implementation.

Concrete repository evidence

At current main revision 1e668906d2e69a9e8ee9aaafc60050a4025d9688:

• result_sets_match converts both outputs to sets.
• test_duplicate_rows_collapsed explicitly asserts that [(1,), (1,), (2,)] equals [(1,), (2,)].

Standalone SQL reproducer:

CREATE TABLE t(x INTEGER);
INSERT INTO t VALUES (1), (1), (2);

• Gold: SELECT x FROM t -> [(1,), (1,), (2,)]
• Prediction: SELECT DISTINCT x FROM t -> [(1,), (2,)]

The current comparator returns a match even though the queries have observably different SQL results. This is inherent to the comparator and does not depend on a particular agent.

Description

Add a versioned/configurable execution-result comparison contract while preserving current unordered set equality as the default. Candidate modes:

• unordered_set — current behavior;
• unordered_multiset — preserve duplicate multiplicity;
• ordered_rows — preserve row order when task metadata declares ordering significant.

Expose the selected mode and normalized comparison diagnostics in verifier output and metrics. Prefer a reusable SQL-result comparison utility if that fits the project architecture.

Design

Potentially touch:

• resources_servers/bird_sql/eval_utils.py;
• resources_servers/bird_sql/app.py;
• resources_servers/bird_sql/configs/bird_sql.yaml;
• focused unit tests with duplicate rows, nulls, ordering, empty results, and unhashable values;
• documentation describing defaults and score comparability.

Unknown modes should fail closed. Existing users should receive equivalent rewards unless they explicitly opt into another mode.

Out of scope

• Changing the default BIRD leaderboard semantics.
• Adding an LLM judge.
• Integrating a particular model, schema-grounding system, retrieval library, or agent.
• Correcting benchmark questions or gold SQL.

Acceptance Criteria

• Existing unordered-set behavior remains the default.
• Multiset mode preserves duplicate multiplicity.
• Ordered behavior is supported or explicitly deferred with task-metadata requirements documented.
• Verifier output records comparator name/version.
• Tests cover duplicates, nulls, empty results, ordering, and execution failures.
• Documentation explains score-comparability implications.

Start with result_sets_match in resources_servers/bird_sql/eval_utils.py and test_duplicate_rows_collapsed in resources_servers/bird_sql/tests/test_app.py, then trace verifier output through app.py and configs/bird_sql.yaml. Done means the current set behavior remains the default, explicitly selected modes and comparator version are recorded, focused tests cover the listed edge cases, and documentation explains score comparability.