a2aproject / a2aproject/a2a-tck

[Feat] Add applicability for prompt-dependent artifact/message scenarios

Aperta
#229 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
50
Fork
40
Merge medio
7g 1h
PR unite (30g)
1

Descrizione

## Environment

- TCK repository: `a2aproject/a2a-tck`
- Tested commit: `5996b79f9cefa6fc390980e383e358a66fb9e49e`
- Level: `must`
- Transports: JSON-RPC and HTTP+JSON
- SUT: an A2A 1.0 narrow travel-search agent that requires exact dates, occupancy and a hard EUR
budget before it may create a priced result

## Reproduction

Run the unchanged official TCK against a conformant narrow agent that returns a typed
`input-required` task for the literal prompt `TCK artifact test`:

```text
uv run python run_tck.py --sut-host --transport jsonrpc --level must -v
uv run python run_tck.py --sut-host --transport http_json --level must -v
```

The same five compatibility tests fail on both transports:

```text
tests.compatibility.core_operations.test_artifacts.TestTextArtifact::test_task_has_text_artifact
tests.compatibility.core_operations.test_artifacts.TestFileArtifact::test_task_has_file_artifact
tests.compatibility.core_operations.test_artifacts.TestFileUrlArtifact::test_task_has_file_url_artifact
tests.compatibility.core_operations.test_artifacts.TestDataArtifact::test_task_has_data_artifact
tests.compatibility.core_operations.test_artifacts.TestMessageResponse::test_returns_message_with_text_part
```

All five tests send the same generic text, `TCK artifact test`. Their generated `messageId` values
use these scenario prefixes: `artifact-text`, `artifact-file`, `artifact-file-url`,
`artifact-data`, and `message-response`.

## Actual behavior

The SUT returns a valid A2A task in `input-required` state with typed missing-field details and a
text fallback. It does not invent travel inventory, dates, a budget, a file, a URL or an unrelated
artifact. Transport, Agent Card, content-type and normal task lifecycle checks pass.

Redacted response fragment, equivalent on both transports:

```json
{
"status": {
"state": "TASK_STATE_INPUT_REQUIRED",
"message": {
"parts": [
{
"data": {
"error": "MISSING_REQUIRED_FIELD",
"message": "exact input is required before searching",
"details": {
"missing": [
"checkin and checkout as exact ISO dates",
"hard EUR budget",
"one- or two-night duration"
]
}
}
}
]
}
},
"artifacts": []
}
```

At the pinned revision, the public test totals are:

- JSON-RPC: 58 passed, 5 failed, 172 skipped;
- HTTP+JSON: 54 passed, 6 failed, 175 skipped.

The additional HTTP+JSON failure is the separately reported streamed-error-body defect in
[issue #225](https://github.com/a2aproject/a2a-tck/issues/225), with a proposed fix in
[PR #226](https://github.com/a2aproject/a2a-tck/pull/226).

## Expected behavior

Protocol conformance should not require a domain-specific agent to fabricate the artifact or direct
message implied by a generic magic prompt. One of these applicability mechanisms would keep the
tests useful without weakening protocol checks:

1. allow the SUT manifest to declare unsupported scenario fixtures;
2. treat a valid `input-required` task as not applicable for these product-semantic scenarios;
3. make artifact/message fixtures opt-in capabilities rather than unconditional MUST expectations;
4. split protocol-shape validation from the prompt-dependent example SUT assertions.

The report does not request a waiver for malformed messages, invalid task states or broken
transports. It requests only a machine-readable applicability path for prompt-dependent product
semantics.

## Evidence hygiene

The minimal reproduction above contains no credential, provider payload, affiliate identifier,
private URL, personal data or customer prompt. Raw reports remain in the private CI artifact store.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

The issue is about the TCK's artifact and message response tests. Look at the failing tests in `tests/compatibility/core_operations/test_artifacts.py`. Understand how the test fixtures (like 'TCK artifact test') are used. The goal is to modify the test framework to allow an SUT to declare unsupported scenarios or to treat a valid `input-required` task as not applicable. Check the SUT manifest structure and the test applicability logic.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
testing-qa
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.