a2aproject / a2aproject/a2a-tck
Tests that record a requirement as failed then xfail: compatibility.json and the exit code disagree
- Langage dominant
- Python
- Étoiles
- 50
- Forks
- 40
- Merge moyen
- 7 j 1 h
- PR mergées (30 j)
- 1
Description
### Summary
Nine tests across three files call `record(..., passed=False)` and then
`pytest.xfail(...)` instead of asserting. The requirement is written into
`compatibility.json` as non-compliant, but pytest reports xfail, so the
suite exits 0. A conformance report can declare a SUT non-compliant on a
MUST or SHOULD requirement while CI reports success.
### Mechanism
`tests/compatibility/_test_helpers.py` — `assert_and_record` records and
then asserts:
```python
passed = not errors
record(collector, req, transport, passed=passed, errors=errors)
assert passed, fail_msg(req, transport, "; ".join(errors))
```
The xfail sites record identically but xfail instead. `record()` accepts
`passed` and `skipped` and has no parameter for a known/expected failure,
so the collector cannot tell the two paths apart.
### Affected sites
- `core_operations/test_data_model.py`: 341, 379 (DM-SERIAL-005)
- `core_operations/test_task_history.py`: 87, 171, 211, 278, 320
(includes CORE-HIST-005 at 278 and CORE-HIST-006 at 320)
- `core_operations/test_requirements.py`: 239, 288
None carries a comment or linked issue explaining why the failure is
expected. `git log -S "pytest.xfail"` traces the DM-SERIAL-005 ones to
their introducing commit in #175.
### Observed
Running the TCK against the reference SUT (`sut/a2a-python`, current main
including #222), DM-SERIAL-005 appears under FAILED REQUIREMENTS:
* DM-SERIAL-005 (jsonrpc, http_json): Server rejected request with
unrecognized fields: Invalid params with no corresponding entry in the pytest failure list, and the run
summary ends `9 failed, 198 passed, 56 skipped, 2 xfailed`.
Separately, @jmesnil reported CORE-HIST-005/006 failing against
a2a-python v1.1.3-11-g35ef52e in #222, while the same requirements pass
in my run. Neither outcome affected the exit code, which makes this class
of divergence hard to notice.
### Question
The right fix depends on why the xfails were written — whether these are
known SDK gaps being tolerated deliberately, or leftovers. Options seem
to be: drop the xfails in favour of `assert_and_record`; add a
`known_failure` flag to `record()` so the report can distinguish; or have
the report generator treat xfailed requirements separately from failed
ones. Happy to send a patch once there's a preferred direction.
Related to #214 and #215 — same underlying theme (the compatibility
report and the test outcomes are computed independently and can
disagree), different mechanism.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.