python-jsonschema / python-jsonschema/jsonschema

unevaluatedProperties error gets raised before more relevant validation errors

Open
#1,132 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement Error Reporting
Dominant language
Python
Stars
5k
Forks
671
Avg merge
1d 1h
Merged PRs (30d)
10

Description

Hello!

I noticed that in schemas using 'allOf' and unevaluatedProperties, when there is an error in a subschema, instead of showing the relevant error, unevaluatedProperties is (sometimes?) shown instead. This makes it really hard to see what went wrong in anything larger than trivial examples. This seems to be related to https://github.com/python-jsonschema/jsonschema/issues/1074, just with a different scenario.

Here is a small example:

import jsonschema
obj = {
    "fridge_min_temp": 10
}
schema = {
  "type": "object",
  "allOf": [
    {
      "properties": {
        "fridge_min_temp": { "type": "integer", "minimum": -20, "maximum": 5 }
      },
      "required": ["fridge_min_temp"]
    }
  ],
  "unevaluatedProperties": False
}
jsonschema.validate(obj, schema)

Which produces the error trace:

ValidationError: Unevaluated properties are not allowed ('fridge_min_temp' was unexpected)

Failed validating 'unevaluatedProperties' in schema:
    {'allOf': [{'properties': {'fridge_min_temp': {'maximum': 5,
                                                   'minimum': -20,
...
     'type': 'object',
     'unevaluatedProperties': False}

On instance:
    {'fridge_min_temp': 10}

If I validate instead using:

jsonschema.Draft202012Validator(schema).validate(obj)

Then the raised error is much more helpful:

ValidationError: 10 is greater than the maximum of 5

Failed validating 'maximum' in schema['allOf'][0]['properties']['fridge_min_temp']:
    {'maximum': 5, 'minimum': -20, 'type': 'integer'}

On instance['fridge_min_temp']:
    10

Seems like this happens because jsonschema.validate is equivalent to raise best_match(jsonschema.Draft202012Validator(schema).iter_errors(obj)) in this case, and the best_match heuristic downgrades any errors coming from 'allOf' and 'oneOf'.
With a more complex schema, I get an unevaluatedProperties error using both approaches.

3 ways of going about it could be:

  1. The suggestion from https://github.com/python-jsonschema/jsonschema/issues/1074 to separately keep track of unevaluated properties and invalid properties, and write different error messages for them.

  2. Add properties to evaluated keys even if there are errors resulting from validator.descend in https://github.com/python-jsonschema/jsonschema/blob/main/jsonschema/_utils.py:

def find_evaluated_property_keys_by_schema(validator, instance, schema):
# ...
  for keyword in ["allOf", "oneOf", "anyOf"]:
    if keyword in schema:
      for subschema in schema[keyword]:
        errs = next(validator.descend(instance, subschema), None)
        if errs is None:
          evaluated_keys += find_evaluated_property_keys_by_schema(
              validator, instance, subschema,
          )

This way unevaluatedProperties will not get raised for properties with subschema errors at all (which shouldn't be a problem, because it'll get raised when those errors are fixed).

  1. Make unevaluatedProperties messages even weaker than "allOf" and "anyOf" related errors in https://github.com/python-jsonschema/jsonschema/blob/main/jsonschema/exceptions.py#L19

I'd be happy to attempt a PR; do you have any thoughts about which approach is best?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the supplied schema with jsonschema.validate and Draft202012Validator.validate, then inspect jsonschema/_utils.py's find_evaluated_property_keys_by_schema and the error-selection logic in jsonschema/exceptions.py. Compare the resulting errors and determine how unevaluatedProperties should avoid masking the more relevant subschema validation error; done means the example reports the maximum violation consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.