a2aproject / a2aproject/a2a-dotnet

Investigate compliance issues with the current a2a-tck

Ouverte
#162 0 commentaires 0 réactions 1 personne assignée Réclamée par @rokonec Voir sur GitHub
Langage dominant
C#
Étoiles
262
Forks
64
Merge moyen
5 j 2 h
PR mergées (30 j)
31

Description

## Description

The https://github.com/a2aproject/a2a-tck has been updated to support v0.3.0 of the specification and many new checks were added. a2a-dotnet is not passing even mandatory set of checks now.

Within this tasks, identify particular a2a-dotnet deficiencies and report individual issues to the backlog that are preventing being compliant with a2a-tck.

Relates to #141.

## Results (WIP)
- [AGENT] Test suite initialization fails because getting agent card expects `preferredTransport` field set. Fix is WIP.
- [FALSE_POSITIVE] `transport_get_agent_card` fails on missing `agent/authenticatedExtendedCard`. This is rather false positive as the test might use unauthenticated agent card. Otherwise, this test is skipped if agent declares push notification support. Fix submitted as https://github.com/a2aproject/a2a-tck/pull/64.
- [FALSE_POSITIVE] Multiple tests from various categories fail on inner `Message` object missing `kind` property. Fix submitted as https://github.com/a2aproject/a2a-tck/pull/63.
- `mandatory`
- `authentication`
- `test_auth_compliance_v030.py`
- ~~[AGENT] `test_security_scheme_structure_compliance` is skipped due to missing `securitySchemes` in agent card.~~ (fixed by #168)
- `test_auth_enforcement.py`
- [FEATURE v0.3.0] `test_authentication_required_when_declared` fails on authentication enforcement.
- [AGENT/FEATURE] `test_invalid_credentials_rejected` fails because invalid authentication request is not rejected.
- ~~[AGENT] `test_authentication_scheme_consistency` fails on missing `security` field on agent card.~~ (fixed by #165)
- `jsonrpc`
- `test_a2a_error_codes_enhanced.py`
- [BUG/FEATURE] `test_push_notification_not_supported_error_32003_enhanced` fails because the tests checks for push notification support and expects apropriate error message when they are declared as not supported. It fails on missing `tasks/pushNotificationConfig/delete` endpoint when they are declared supported.
- [IMPROVEMENT] `test_unsupported_operation_error_32004_enhanced` is skipped because agent do not trigger error when sending unsupported format in `acceptedOutputModes` field of `configuration` param to `message/send` endpoint.
- [IMPROVEMENT] `test_content_type_not_supported_error_32005_enhanced` is skipped because agent do not trigger error when sending file part with unsupported MIME type to `message/send` endpoint.
- [FALSE_NEGATIVE] `test_invalid_agent_response_error_32006_enhanced` is skipped because the test fails to produce invalid request.
- `test_a2a_error_codes.py`
- [IMPROVEMENT] `test_push_notification_not_supported_error_32003` is skipped when agent card does not declare push notifications support while `tasks/pushNotificationConfig/set` endpoint is implemented.
- [IMPROVEMENT] `test_unsupported_operation_error_32004` is skipped because agent do not trigger error when sending unsupported format in `acceptedOutputModes` field of `configuration` param to `message/send` endpoint.
- [IMPROVEMENT] `test_content_type_not_supported_error_32005` is skipped because agent do not trigger error when sending file part with unsupported MIME type to `message/send` endpoint.
- `test_json_rpc_compliance.py`
- [BUG] `test_rejects_invalid_json_rpc_requests` fails becase agent return error -32600 instead -32602 when `message/send` RPC request params contains "" as object key.
- [BUG] `test_rejects_invalid_params` fails becase agent return error -32600 instead -32602 when `message/send` RPC request params is string instead of expected object.
- `protocol`
- `test_a2a_v030_new_methods.py`
- [FALSE_NEGATIVE/FEATURE] `test_authenticated_extended_card_method_exists` fails because of wrong implementation of Advanced Agent Card retrieval on `a2a-tck` side, issue submitted as https://github.com/a2aproject/a2a-tck/issues/61, fix submitted as https://github.com/a2aproject/a2a-tck/pull/64. Then implementation of Advanced Agent Card on `a2a-dotnet` side will be needed.
- [FALSE_NEGATIVE/FEATURE] `test_authenticated_extended_card_without_auth` same as above.
- [FALSE_NEGATIVE/FEATURE] `test_authenticated_extended_card_with_auth` same as above.
- [AGENT] `test_tasks_list_method_availability` is skipped because of check for JSON-RPC transport.
- [AGENT] `test_tasks_list_with_existing_tasks` is skipped because of check for JSON-RPC transport.
- [AGENT/FEATURE] `test_grpc_specific_features` is skipped becase transport is not gRPC.
- [AGENT] `test_rest_specific_features` is skipped becase transport is not REST.
- `test_a2a_v030_transport_compliance.py`
- [FEATURE] `test_multi_transport_method_mapping` is skipped because not all transports are supported.
- `test_agent_card_mandatory.py`
- [FALSE_POSITIVE] `test_agent_card_skills_mandatory` fails because the test mandates skills array not being empty in the agent card. Reported as https://github.com/a2aproject/a2a-tck/issues/65.
- `test_extended_agent_card.py`
- [FEATURE v0.3.0] `test_extended_agent_card_endpoint_exists` fails because `agent/getAuthenticatedExtendedCard` method is not implemented.
- [FEATURE v0.3.0] `test_extended_agent_card_authentication_required` fails because `agent/getAuthenticatedExtendedCard` method is not implemented.
- [FEATURE v0.3.0] `test_extended_agent_card_response_format` is skipped because `agent/getAuthenticatedExtendedCard` method is not implemented.
- `test_message_send_method.py`
- [BUG] `test_message_send_invalid_params` fails becase agent returns error -32600 instead of -32602 when `message/send` RPC request is missing required properties of message object.
- `quality`
- `test_error_validation_enhanced.py`
- [BUG] `test_invalid_method_error_validation` failse because agent returns -32603 instead of -32601 when non-existent method is requested.
- `securtity`
- `test_agent_card_security.py`
- [A2A_BUG/FALSE_POSITIVE/BUG] `test_public_agent_card_access_control` fails on miss alignment what should be the `url` property in `AgentCard`. The test tries to access the `url` directly to get the card wich is wrong because it should be at least `/.well-known/agent-card.json`. On the other hand, the `SpecComplianceAgent` returns `http://localhost:5048/speccompliance` in the `url` which is also wrong because `http://localhost:5048/speccompliance/.well-known/agent-card.json` is not a valid location for the card. This is basically another consequence in the design of how agent cards should be served in the specification.
- [A2A_BUG/FALSE_POSITIVE/FEATURE] `test_extended_card_access_controls` fails because the tests construct the extended authenticated agent card URL wrong but there is no clarity on where is should be in the specification either.
- [A2A_BUG/FALSE_NEGATIVE/FEATURE] `test_authentication_scheme_validation` tests passes but there are validation errors with the same cause as `test_extended_card_access_controls`.
- [FALSE_POSITIVE/AGENT] `test_sensitive_information_protection` failse because the test checks for certain patterns in the public agent card on deserialized structure converted to string which contains fields for secrects even if they were not present in the agent response. Nevertheless, the `SpecComplianceAgent` does not have any special handling that no credentials are exposed on public card and extended authenticated card is not supported.
- `test_certificate_validation.py`
- [UNDECICIVE] All tests in this file are skipped because they require agent running on HTTPS with a domain name.
- `test_in_task_authentication.py`
- [UNDECICIVE/AGENT/FEATURE] `test_in_task_authentication_workflow` is skipped becuase the test is trying to authenticate using LLM prompt submitted as task while the `SpecComplianceAgent` has no LLM.
- [FALSE_NEGATIVE/AGENT/FEATURE] `test_authentication_challenge_headers` passes even if it tries to request non-existing task and check if auhentication was enforced. This is not implemented both in the `SpecComplianceAgent` and the library.
- [FEATURE] `test_invalid_authentication_handling` fails because agent do not respond with correct errors on requests with ivalid authentication.
- [AGENT/FEATURE] `test_auth_state_transitions` is skipped because `SpecComplianceAgent` do not implement tasks state management and authentication.
- `test_tls_configuration_enhanced.py`
- [UNDECICIVE/FEATURE] All tests in this file are skipped because they require agent running on HTTPS with a domain name and TLS support.
- `test_transport_security.py`
- [UNDECICIVE/FEATURE] All tests in this file are skipped because they require agent running on HTTPS with a domain name and TLS support.
- `transport`
- `test_multi_transport_equivalence.py`
- [FALSE_POSITIVE/FALSE_NEGATIVE] Tests in this file fails because they are unfinished / regressed. Some of them pass but should fail for the same reason.
- `unit`
- [FALSE_POSITIVE] Some unit test fails on usage of removed `send_json_rpc` method of `SUTClient`. Fix submitted as https://github.com/a2aproject/a2a-tck/pull/66.
- [FALSE_POSITIVE] `test_transport_error_with_original` fails because of signature of `TransportError` has changed. Fix submitted as https://github.com/a2aproject/a2a-tck/pull/67.
- [FALSE_POSITIVE] `test_get_task_interface` and `test_cancel_task_interface` fail on invalid assert of task ID. Fix submitted as https://github.com/a2aproject/a2a-tck/pull/68.

## Explanation

- [A2A_BUG]: Issue in A2A specification.
- [AGENT]: Issue in `SpecComplianceAgent` that can be resolved without change to `a2a-dotnet`
- [BUG]: Issue in `a2a-dotnet` in already implemented feature.
- [FALSE_POSITIVE]: Issue in `a2a-tck` where test is failing but shouln't.
- [FALSE_NEGATIVE]: Issue in `a2a-tck` where test is passing but shouln't.
- [FEATURE ]: Unimplemented feature in `a2a-dotnet` for the specified protocol version.
- [FEATURE]: Unimplemented feature in `a2a-dotnet`, the protocol version is unknown.
- [IMPROVEMENT]: Not a bug but improvement in `a2a-dotnet` can resolve.
- [UNDECICIVE]: The test cannot be run for a reason.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.