a2aproject / a2aproject/a2a-tck
CORE-SEND-003 and CORE-MULTI-002a omit expected_error, so both assert the inverse of the MUST they cite
- 主要語言
- Python
- 星號
- 50
- 分支
- 40
- 平均合併
- 7 天 1 小時
- 30 天內合併 PR
- 1
描述
## What happened?
Two MUST-level `RequirementSpec` entries omit the `expected_error=` field, so the generic validator falls through to its success path. Both tests therefore **assert the exact opposite of the requirement they cite**, and cannot be passed by a spec-conformant agent.
Found at commit `29063fe95e903cddac5d8ff811ab94df1ad6ef86` (tag `1.0.0.alpha2`) while running the suite against our HTTP+JSON implementation.
### 1. CORE-SEND-003 — asserts success on an unsupported media type
`tck/requirements/core_operations.py:95-118` declares:
```python
RequirementSpec(
id="CORE-SEND-003",
title="SendMessage returns ContentTypeNotSupportedError for unsupported media",
level=RequirementLevel.MUST,
description=(
"A Media Type provided in the request's message parts that is not "
"supported by the agent MUST result in ContentTypeNotSupportedError."
),
expected_behavior="ContentTypeNotSupportedError returned",
# <-- no expected_error=
sample_input={"message": {"parts": [{"raw": "dGNr", "mediaType": "application/x-unsupported-tck-type"}], ...}},
)
```
Compare its immediate neighbour CORE-SEND-002 at `:90`, which does wire one:
```python
expected_behavior="UnsupportedOperationError returned for terminal task",
expected_error=UNSUPPORTED_OPERATION_ERROR,
```
In `tests/compatibility/core_operations/test_requirements.py:87-130`, `_validate_response` branches on that field:
```python
if requirement.expected_error is not None:
return validate_expected_error(response, transport, requirement.expected_error)
...
if not response.success:
errors.append(f"Operation failed: {response.error}")
return errors
```
With `expected_error` unset, the test asserts `response.success is True`. So a MUST test titled *"returns ContentTypeNotSupportedError"* **requires the agent not to return it**. An implementation that correctly rejects `application/x-unsupported-tck-type` fails; one that ignores media types and echoes passes.
`CONTENT_TYPE_NOT_SUPPORTED_ERROR` is defined at `tck/requirements/base.py:293` and registered at `:377`, but is never referenced by any `RequirementSpec`. Only four specs wire `expected_error` at all (`core_operations.py:90`, `:231`, `:376`, `:642`) — none of them CORE-SEND-003.
We believe this passes today only because the reference SUT's default branch (`sut/a2a-python/sut_agent.py:157-162`) completes on any input regardless of media type, so the gap is invisible from inside this repo.
### 2. CORE-MULTI-002a — same defect
`tck/requirements/core_operations.py:585-608`:
```python
RequirementSpec(
id="CORE-MULTI-002a",
title="Agent rejects unacceptable client-provided contextId",
level=RequirementLevel.MUST,
# <-- no expected_error=
)
```
A requirement whose title is *"Agent rejects…"* likewise asserts success, so it tests the inverse of its title.
## Suggested fix
Wire the existing bindings:
- CORE-SEND-003 → `expected_error=CONTENT_TYPE_NOT_SUPPORTED_ERROR`
- CORE-MULTI-002a → the appropriate rejection binding (`INVALID_ARGUMENT`-family; we did not want to guess your intended mapping here)
A cheap regression guard: assert at collection time that any `RequirementSpec` whose `expected_behavior` matches `/Error returned/` (or whose title starts with "Agent rejects") also sets `expected_error`. That single check would have caught both.
## Relevant log output
Running against an HTTP+JSON implementation that returns `ContentTypeNotSupportedError` for unsupported media:
```
✗ CORE-SEND-003 (http_json): Operation failed: [400] ContentTypeNotSupportedError
AssertionError: CORE-SEND-003 [SendMessage returns ContentTypeNotSupportedError for unsupported media]
failed on http_json (see specification/specification.md#311-send-message)
```
The agent did exactly what the requirement's `description` demands, and the test recorded it as a MUST failure.
## Notes
Happy to send a PR wiring CORE-SEND-003 to `CONTENT_TYPE_NOT_SUPPORTED_ERROR` plus the collection-time guard, if that direction is welcome. We would rather confirm the intended mapping for CORE-MULTI-002a with you first than guess it.
Related, previously filed: #201.
貢獻指南
評估
這個 Issue 還沒有評估資料。