Investigate if we can improve Pydantic's error messages for user consumption

Open

#13,337 opened on Dec 20, 2023 0 comments 0 reactions 0 assignees View on GitHub

A-Validation T-Task Z-Dev-Wishlist

Dominant language: Python
Stars: 4.6k
Forks: 600
Avg merge: 5d 22h
Merged PRs (30d): 51

Description

This issue has been migrated from [#13337](https://github.com/matrix-org/synapse/issues/13337).

---

If anyone wants to see the kind of error messages Pydantic produces, try running [this test case](https://github.com/matrix-org/synapse/blob/bfa3ae3d3bb158fa022c88a63561d60994c3e1df/tests/config/test_oidc2.py#L58-L65) on my other pydantic experiment branch.

I think the name of the model isn't user-relevant. The "error context" (stuff in brackets) also loks like noise to an end-user. But we should be able to inspect the `ValidationError` and format it as we please.

```
/home/dmr/.cache/pypoetry/virtualenvs/matrix-synapse-7yaa6cKe-py3.10/bin/python /home/dmr/.local/share/JetBrains/Toolbox/apps/PyCharm-C/ch-0/221.5787.24/plugins/python-ce/helpers/pycharm/_jb_unittest_runner.py --path /home/dmr/workspace/synapse-2/tests/config/test_oidc2.py
Testing started at 19:38 ...
Launching unittests with arguments python -m unittest /home/dmr/workspace/synapse-2/tests/config/test_oidc2.py in /home/dmr/workspace/synapse-2/tests/config

Ran 13 tests in 0.033s

Process finished with exit code 0

1 validation error for OIDCProviderModel
attribute_requirements
value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements
value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements -> 0
value is not a valid dict (type=type_error.dict)

1 validation error for OIDCProviderModel
attribute_requirements
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> attribute
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
field required (type=value_error.missing)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> attribute
field required (type=value_error.missing)

2 validation errors for OIDCProviderModel
attribute_requirements -> 0 -> attribute
field required (type=value_error.missing)
attribute_requirements -> 0 -> value
field required (type=value_error.missing)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> answer
extra fields not permitted (type=value_error.extra)

1 validation error for OIDCProviderModel
client_auth_method
value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[, , ])

1 validation error for OIDCProviderModel
client_auth_method
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
discover
value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
discover
value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
authorization_endpoint
authorization_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
token_endpoint
token_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
idp_brand
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
expected string or bytes-like object (type=type_error)

1 validation error for OIDCProviderModel
idp_icon
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
mxc URI '' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
mxc URI 'notaurl' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
mxc URI 'https://example.com' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
mxc URI 'mxc://mxc://mxc://' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_id
field required (type=value_error.missing)

1 validation error for OIDCProviderModel
idp_id
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
idp_id
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
ensure this value has at least 1 characters (type=value_error.any_str.min_length; limit_value=1)

1 validation error for OIDCProviderModel
idp_id
ensure this value has at most 250 characters (type=value_error.any_str.max_length; limit_value=250)

1 validation error for OIDCProviderModel
idp_id
string does not match regex "^[A-Za-z0-9._~-]+$" (type=value_error.str.regex; pattern=^[A-Za-z0-9._~-]+$)

1 validation error for OIDCProviderModel
idp_id
ensure this value has at most 250 characters (type=value_error.any_str.max_length; limit_value=250)

1 validation error for OIDCProviderModel
issuer
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
issuer
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
field required (type=value_error.missing)

2 validation errors for LegacyOIDCProviderModel
idp_id
str type expected (type=type_error.str)
idp_name
str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
none is not an allowed value (type=type_error.none.not_allowed)
idp_name
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes
value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes -> 0
none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes -> 0
str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
userinfo_endpoint
userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

```

_Originally posted by @DMRobertson in https://github.com/matrix-org/synapse/issues/13188#issuecomment-1175382843_

Contributor guide

Open the contributing guide

Research direction

Run tests/config/test_oidc2.py and review the Pydantic ValidationError output shown in the issue. The payload does not identify an implementation file or a decided formatting approach; done would require an agreed user-facing error format and corresponding validation coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack: python
Domain: authentication, backend
Issue type: Feature
Difficulty: 5/5
Estimated time: Over a week
Activity status: Stale
Clarity: Needs clarification
Newbie friendliness: 25/100

Investigate if we can improve Pydantic's error messages for user consumption

Description

Contributor guide

Research direction

Assessment

Get new issues in your inbox