python-jsonschema / python-jsonschema/jsonschema
unevaluatedProperties error gets raised before more relevant validation errors
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 671
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 10
Description
Hello!
I noticed that in schemas using 'allOf' and unevaluatedProperties, when there is an error in a subschema, instead of showing the relevant error, unevaluatedProperties is (sometimes?) shown instead. This makes it really hard to see what went wrong in anything larger than trivial examples. This seems to be related to https://github.com/python-jsonschema/jsonschema/issues/1074, just with a different scenario.
Here is a small example:
import jsonschema
obj = {
"fridge_min_temp": 10
}
schema = {
"type": "object",
"allOf": [
{
"properties": {
"fridge_min_temp": { "type": "integer", "minimum": -20, "maximum": 5 }
},
"required": ["fridge_min_temp"]
}
],
"unevaluatedProperties": False
}
jsonschema.validate(obj, schema)
Which produces the error trace:
ValidationError: Unevaluated properties are not allowed ('fridge_min_temp' was unexpected)
Failed validating 'unevaluatedProperties' in schema:
{'allOf': [{'properties': {'fridge_min_temp': {'maximum': 5,
'minimum': -20,
...
'type': 'object',
'unevaluatedProperties': False}
On instance:
{'fridge_min_temp': 10}
If I validate instead using:
jsonschema.Draft202012Validator(schema).validate(obj)
Then the raised error is much more helpful:
ValidationError: 10 is greater than the maximum of 5
Failed validating 'maximum' in schema['allOf'][0]['properties']['fridge_min_temp']:
{'maximum': 5, 'minimum': -20, 'type': 'integer'}
On instance['fridge_min_temp']:
10
Seems like this happens because jsonschema.validate is equivalent to raise best_match(jsonschema.Draft202012Validator(schema).iter_errors(obj)) in this case, and the best_match heuristic downgrades any errors coming from 'allOf' and 'oneOf'.
With a more complex schema, I get an unevaluatedProperties error using both approaches.
3 ways of going about it could be:
-
The suggestion from https://github.com/python-jsonschema/jsonschema/issues/1074 to separately keep track of unevaluated properties and invalid properties, and write different error messages for them.
-
Add properties to evaluated keys even if there are errors resulting from
validator.descendin https://github.com/python-jsonschema/jsonschema/blob/main/jsonschema/_utils.py:
def find_evaluated_property_keys_by_schema(validator, instance, schema):
# ...
for keyword in ["allOf", "oneOf", "anyOf"]:
if keyword in schema:
for subschema in schema[keyword]:
errs = next(validator.descend(instance, subschema), None)
if errs is None:
evaluated_keys += find_evaluated_property_keys_by_schema(
validator, instance, subschema,
)
This way unevaluatedProperties will not get raised for properties with subschema errors at all (which shouldn't be a problem, because it'll get raised when those errors are fixed).
- Make
unevaluatedPropertiesmessages even weaker than "allOf" and "anyOf" related errors in https://github.com/python-jsonschema/jsonschema/blob/main/jsonschema/exceptions.py#L19
I'd be happy to attempt a PR; do you have any thoughts about which approach is best?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the supplied schema with jsonschema.validate and Draft202012Validator.validate, then inspect jsonschema/_utils.py's find_evaluated_property_keys_by_schema and the error-selection logic in jsonschema/exceptions.py. Compare the resulting errors and determine how unevaluatedProperties should avoid masking the more relevant subschema validation error; done means the example reports the maximum violation consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100