python-jsonschema / python-jsonschema/jsonschema

Performance degradation moving from Python 3.10 to 3.12

Open
#1,304 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5k
Forks
671
Avg merge
1d 1h
Merged PRs (30d)
10

Description

I was running some timing today and noticed it was taking longer to run the schema validation under Python 3.12 than it was under Python 3.10. 16MB SBOM generated by Syft. About 45 seconds under 3.10, about 48 seconds under 3.12. Sadly, I'm not allowed to share the SBOM outside of my company. But, I can share my data. :)

Under profiling, the time differences are even more stark. 135 seconds under 3.10, 179 seconds under 3.12

Script:

import os
import sys

from cyclonedx.schema import SchemaVersion
from cyclonedx.validation.json import JsonStrictValidator

import cProfile
import pstats
from pstats import SortKey


def main():
    my_json_validator = JsonStrictValidator(SchemaVersion.V1_6)
    validation_errors = my_json_validator.validate_str(open('syft_sbom.json').read())
    if validation_errors:
        print('JSON invalid', 'ValidationError:', repr(validation_errors), sep='\n', file=sys.stderr)
        sys.exit(2)
    print('JSON valid')

v = sys.version_info
stat_file = os.path.expanduser(f"~/tmp/validation_{'_'.join([str(v.major), str(v.minor), str(v.micro)])}.data")
cProfile.run('main()', stat_file)
pstats.Stats(stat_file).sort_stats(SortKey.TIME, SortKey.CUMULATIVE).print_stats(50)
os.unlink(stat_file)

Versions installed:

Python 3.10

$ pip3 list
Package                   Version
------------------------- --------------
arrow                     1.3.0
attrs                     24.2.0
boolean.py                4.0
cyclonedx-python-lib      7.6.1
defusedxml                0.7.1
fqdn                      1.5.1
idna                      3.10
isoduration               20.11.0
jsonpointer               3.0.0
jsonschema                4.23.0
jsonschema-specifications 2023.12.1
license-expression        30.3.1
packageurl-python         0.15.6
pip                       24.2
py-serializable           1.1.2
python-dateutil           2.9.0.post0
referencing               0.35.1
rfc3339-validator         0.1.4
rfc3987                   1.3.8
rpds-py                   0.20.0
setuptools                74.1.2
six                       1.16.0
sortedcontainers          2.4.0
types-python-dateutil     2.9.0.20240906
uri-template              1.3.0
webcolors                 24.8.0

Python 3.12. Identical, except for the lack of setuptools.

Package                   Version
------------------------- --------------
arrow                     1.3.0
attrs                     24.2.0
boolean.py                4.0
cyclonedx-python-lib      7.6.1
defusedxml                0.7.1
fqdn                      1.5.1
idna                      3.10
isoduration               20.11.0
jsonpointer               3.0.0
jsonschema                4.23.0
jsonschema-specifications 2023.12.1
license-expression        30.3.1
packageurl-python         0.15.6
pip                       24.2
py-serializable           1.1.2
python-dateutil           2.9.0.post0
referencing               0.35.1
rfc3339-validator         0.1.4
rfc3987                   1.3.8
rpds-py                   0.20.0
six                       1.16.0
sortedcontainers          2.4.0
types-python-dateutil     2.9.0.20240906
uri-template              1.3.0
webcolors                 24.8.0

Timing under Python 3.10

Tue Oct  1 14:28:45 2024    /Users/tek30584/tmp/validation_3_10_15.data

         1168063553 function calls (1117721469 primitive calls) in 135.435 seconds

   Ordered by: internal time, cumulative time
   List reduced from 228 to 50 due to restriction <50>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
105980718/56440480   28.444    0.000  123.782    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:127(equal)
 56195901   27.838    0.000   54.892    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:106(_mapping_equal)
333189549   23.089    0.000   47.820    0.000 {built-in method builtins.isinstance}
168862378   13.585    0.000   24.731    0.000 /opt/homebrew/Cellar/python@3.10/3.10.15/Frameworks/Python.framework/Versions/3.10/lib/python3.10/abc.py:117(__instancecheck__)
168862378   11.145    0.000   11.145    0.000 {built-in method _abc._abc_instancecheck}
 99080476   10.389    0.000   17.554    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:112(<genexpr>)
        2    5.509    2.755  129.263   64.631 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:156(uniq)
 49551487    4.192    0.000   19.470    0.000 {built-in method builtins.all}
112424250/112423479    3.625    0.000    3.625    0.000 {built-in method builtins.len}
 50463582    1.743    0.000    1.743    0.000 {method 'items' of 'dict' objects}
 540652/7    0.808    0.000  135.364   19.338 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:397(descend)
   541226    0.526    0.000    1.491    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:340(evolve)
   137336    0.325    0.000    0.922    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:249(pointer)
   916230    0.303    0.000    0.446    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_types.py:90(is_type)
   541227    0.293    0.000    0.650    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:283(__attrs_post_init__)
  1081273    0.230    0.000    0.313    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_legacy_keywords.py:9(ignore_ref_siblings)
   402084    0.204    0.000    0.693    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:282(type)
   541227    0.172    0.000    0.172    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:294(<listcomp>)
   676801    0.166    0.000    0.297    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:160(create_resource)
  3497847    0.160    0.000    0.160    0.000 {method 'get' of 'dict' objects}
   916230    0.149    0.000    0.595    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:453(is_type)
   541227    0.143    0.000    0.793    0.000 <attrs generated init jsonschema.validators.create.<locals>.Validator>:1(__init__)
 116122/1    0.138    0.000  135.365  135.365 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:290(properties)
   137920    0.136    0.000    1.481    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:643(lookup)
   676801    0.132    0.000    0.132    0.000 <attrs generated init referencing._core.Resource>:1(__init__)
   539463    0.125    0.000    0.175    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/jsonschema.py:53(_legacy_dollar_id)
   274672    0.109    0.000    0.245    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/jsonschema.py:338(maybe_in_subresource)
   539463    0.107    0.000    0.282    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:222(id)
  2311156    0.106    0.000    0.106    0.000 {built-in method builtins.getattr}
   539462    0.093    0.000    0.375    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:690(in_subresource)
   804166    0.093    0.000    0.368    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:285(<genexpr>)
   137923    0.089    0.000    0.208    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/attr/_funcs.py:397(evolve)
137920/10603    0.066    0.000    6.044    0.001 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:274(ref)
   541226    0.066    0.000    0.071    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:1335(validator_for)
   584178    0.066    0.000    0.092    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_types.py:48(is_object)
   137920    0.062    0.000    0.119    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:405(get_or_retrieve)
   117312    0.061    0.000    0.295    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:36(additionalProperties)
   117312    0.060    0.000    0.085    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:69(find_additional_properties)
   402083    0.057    0.000    0.072    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_utils.py:95(ensure_list)
   137920    0.056    0.000    0.266    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/referencing/_core.py:706(_evolve)
        4    0.055    0.014    0.055    0.014 /opt/homebrew/Cellar/python@3.10/3.10.15/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py:343(raw_decode)
   137923    0.055    0.000    0.084    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/attr/_make.py:1699(fields)
   137920    0.054    0.000    1.536    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/validators.py:460(_validate_reference)
   402083    0.048    0.000    0.393    0.000 {built-in method builtins.any}
  12449/2    0.040    0.000    6.101    3.051 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_legacy_keywords.py:124(items_draft6_draft7_draft201909)
   137921    0.036    0.000    0.036    0.000 <attrs generated init referencing._core.Resolver>:1(__init__)
   294105    0.035    0.000    0.046    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_types.py:52(is_string)
   563360    0.035    0.000    0.035    0.000 {method 'replace' of 'str' objects}
   117310    0.032    0.000    0.106    0.000 /Users/tek30584/programming/merge_poc/.venv.python310/lib/python3.10/site-packages/jsonschema/_keywords.py:304(required)
   137920    0.029    0.000    0.029    0.000 <attrs generated init referencing._core.Retrieved>:1(__init__)

Timing under 3.12

Tue Oct  1 14:33:15 2024    /Users/tek30584/tmp/validation_3_12_6.data

         1167490383 function calls (1117148299 primitive calls) in 179.302 seconds

   Ordered by: internal time, cumulative time
   List reduced from 228 to 50 due to restriction <50>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 56195901   37.214    0.000   75.522    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:106(_mapping_equal)
333186659   34.868    0.000   66.689    0.000 {built-in method builtins.isinstance}
105980718/56440480   33.197    0.000  166.354    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:127(equal)
168862378   16.584    0.000   31.821    0.000 <frozen abc>:117(__instancecheck__)
168862378   15.237    0.000   15.237    0.000 {built-in method _abc._abc_instancecheck}
 99080476   13.222    0.000   21.989    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:112(<genexpr>)
 49551487    6.067    0.000   23.560    0.000 {built-in method builtins.all}
112424251/112423480    5.896    0.000    5.897    0.000 {built-in method builtins.len}
        2    5.838    2.919  172.158   86.079 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:156(uniq)
 50463582    4.473    0.000    4.473    0.000 {method 'items' of 'dict' objects}
 540652/7    0.900    0.000  179.232   25.605 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:397(descend)
   541226    0.597    0.000    1.556    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:340(evolve)
   137336    0.364    0.000    1.120    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:249(pointer)
  3495511    0.348    0.000    0.348    0.000 {method 'get' of 'dict' objects}
   916230    0.329    0.000    0.509    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_types.py:90(is_type)
   541227    0.307    0.000    0.588    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:283(__attrs_post_init__)
   402084    0.279    0.000    0.855    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:282(type)
  1081273    0.273    0.000    0.457    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_legacy_keywords.py:9(ignore_ref_siblings)
   676801    0.218    0.000    0.406    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:160(create_resource)
  2311156    0.206    0.000    0.206    0.000 {built-in method builtins.getattr}
   676801    0.187    0.000    0.187    0.000 <attrs generated init referencing._core.Resource>:1(__init__)
   916230    0.170    0.000    0.679    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:453(is_type)
 116122/1    0.141    0.000  179.232  179.232 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:290(properties)
   539463    0.132    0.000    0.213    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/jsonschema.py:53(_legacy_dollar_id)
   137920    0.131    0.000    1.733    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:643(lookup)
   804166    0.121    0.000    0.422    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:285(<genexpr>)
   539463    0.113    0.000    0.325    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:222(id)
   541227    0.111    0.000    0.699    0.000 <attrs generated init jsonschema.validators.create.<locals>.Validator>:1(__init__)
   274672    0.102    0.000    0.262    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/jsonschema.py:338(maybe_in_subresource)
   539462    0.094    0.000    0.419    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:690(in_subresource)
   137923    0.088    0.000    0.222    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/attr/_funcs.py:397(evolve)
   584178    0.078    0.000    0.117    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_types.py:48(is_object)
   117312    0.069    0.000    0.360    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:36(additionalProperties)
137920/10603    0.068    0.000    7.010    0.001 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_keywords.py:274(ref)
   137920    0.066    0.000    0.145    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:405(get_or_retrieve)
   117312    0.065    0.000    0.113    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:69(find_additional_properties)
   402083    0.065    0.000    0.089    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_utils.py:95(ensure_list)
   402083    0.065    0.000    0.439    0.000 {built-in method builtins.any}
   549356    0.065    0.000    0.065    0.000 {method 'replace' of 'str' objects}
   137920    0.063    0.000    0.287    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/referencing/_core.py:706(_evolve)
   541226    0.057    0.000    0.060    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:1335(validator_for)
        4    0.056    0.014    0.056    0.014 /opt/homebrew/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/json/decoder.py:343(raw_decode)
   541226    0.055    0.000    0.055    0.000 {method 'setdefault' of 'dict' objects}
   137923    0.054    0.000    0.087    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/attr/_make.py:1699(fields)
   137920    0.048    0.000    1.782    0.000 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/validators.py:460(_validate_reference)
   137921    0.047    0.000    0.047    0.000 <attrs generated init referencing._core.Resolver>:1(__init__)
  12449/2    0.042    0.000    7.073    3.537 /Users/tek30584/programming/merge_poc/.venv.python312/lib/python3.12/site-packages/jsonschema/_legacy_keywords.py:124(items_draft6_draft7_draft201909)
   275845    0.041    0.000    0.041    0.000 {method 'startswith' of 'str' objects}
   137920    0.040    0.000    0.040    0.000 <attrs generated init referencing._core.Retrieved>:1(__init__)
   137920    0.039    0.000    0.039    0.000 {method 'get' of 'rpds.HashTrieMap' objects}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied profiling script with a representative 16MB SBOM and compare Python 3.10 with 3.12. Read the JsonStrictValidator path and the jsonschema/_utils.py functions highlighted by the profile, especially equal and _mapping_equal. Done means the validation slowdown is explained and the Python 3.12 regression is addressed without changing validation results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.