plotly / plotly/plotly.py

Allow disabling lookup of missing prop names

Ouverte
#4,100 3 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

feature P3
Langage dominant
Python
Étoiles
18.8k
Forks
2.8k
Merge moyen
16 h 26 min
PR mergées (30 j)
21

Description

When running a large update on a plot, sometimes the time is dominated by the lookup for missing names, even when this lookup serves no purpose, such as in plotly.subplots.make_subplots where new axes are being added by the plotly library itself, and finding "nearby" names is not helpful when all that is needed is a check if the name already exists.

In plotly.basedatatypes.BaseDatatype._perform_update, for each key to be updated it calls _check_path_in_prop_tree, but then ignores the error that is generated if the plotly_obj is a BaseLayoutType and the key matches one of the _subplot_re_match keywords, and just adds the key to the plotly_obj instead.

This call to _check_path_in_prop_tree is expensive when

  • there are a large number of defined keys in the layout (such as when there are a large number of subplots being initialized) and
  • the path is not yet in the prop tree

When those conditions are met, the prop lookup check_path_in_prop_tree calls BasePlotlyType.__getitem__, which then calls BasePlotlyType._raise_on_invalid_property_error, which then calls _plotly_utils.utils.find_closest_string for each new property being added. find_closest_string uses a levenshtein lookup and this takes geometrically more time the more keys there are to compare, which is the root of the problem.

If the lookup could be disabled in _raise_on_invalid_property_error, that would help, but there is a problem in that _check_path_in_prop_tree is using an implicit __getitem__ on the plotly_obj, here

The issue can be resolved by disabling the lookup completely, as shown in the second example notebook below, where _plotly_utils.utils is monkey-patched so that find_closest_string always raises an error, but this doesn't seem like the best way to handle it.

Example one: without disabling lookup

import plotly.subplots
import cProfile
import pstats
from pstats import SortKey

cProfile.run('fig = plotly.subplots.make_subplots(rows=20, cols=20)', 'restats')
p = pstats.Stats('restats')
p.sort_stats(SortKey.CUMULATIVE).print_stats(20)

Mon Mar 13 15:24:56 2023    restats

         55179619 function calls (54968062 primitive calls) in 18.368 seconds

   Ordered by: cumulative time
   List reduced from 868 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    150/1    0.000    0.000   18.368   18.368 {built-in method builtins.exec}
        1    0.000    0.000   18.368   18.368 <string>:1(<module>)
        1    0.000    0.000   18.368   18.368 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/subplots.py:7(make_subplots)
        1    0.002    0.002   18.368   18.368 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/_subplots.py:45(make_subplots)
        1    0.000    0.000   18.043   18.043 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/graph_objs/_figure.py:736(update_layout)
        1    0.000    0.000   18.043   18.043 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:1378(update_layout)
        1    0.000    0.000   18.043   18.043 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:5096(update)
    802/2    0.010    0.000   17.536    8.768 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:3841(_perform_update)
19302/14494    0.065    0.000   17.219    0.001 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:4659(__getitem__)
4893/4891    0.018    0.000   17.209    0.004 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:159(_check_path_in_prop_tree)
14498/14494    0.015    0.000   17.096    0.001 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:5828(__getitem__)
      798    0.005    0.000   16.890    0.021 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:5047(_ret)
      798    0.001    0.000   16.879    0.021 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/_plotly_utils/utils.py:449(find_closest_string)
     1600    0.176    0.000   16.878    0.011 {built-in method builtins.sorted}
   390621    0.099    0.000   16.702    0.000 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/_plotly_utils/utils.py:450(_key)
470776/390621   11.539    0.000   16.603    0.000 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/_plotly_utils/utils.py:430(levenshtein)
 24961225    3.540    0.000    3.540    0.000 {built-in method builtins.min}
 24971843    1.419    0.000    1.419    0.000 {method 'append' of 'list' objects}
        1    0.000    0.000    0.507    0.507 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/contextlib.py:139(__exit__)
        2    0.000    0.000    0.507    0.253 {built-in method builtins.next}


<pstats.Stats at 0x7faae67616f0>

Example two: disabling lookup

## Patch the library to disable levenshtein lookup for missing strings.
def disable_find_closest_string(string, strings):
    raise ValueError()

import _plotly_utils.utils
_plotly_utils.utils.find_closest_string = disable_find_closest_string

import plotly.subplots
import cProfile
import pstats
from pstats import SortKey

cProfile.run('fig = plotly.subplots.make_subplots(rows=20, cols=20)', 'restats')
p = pstats.Stats('restats')
p.sort_stats(SortKey.CUMULATIVE).print_stats(20)

Mon Mar 13 15:25:05 2023    restats

         2674042 function calls (2542640 primitive calls) in 1.498 seconds

   Ordered by: cumulative time
   List reduced from 866 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    150/1    0.000    0.000    1.498    1.498 {built-in method builtins.exec}
        1    0.000    0.000    1.498    1.498 <string>:1(<module>)
        1    0.000    0.000    1.498    1.498 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/subplots.py:7(make_subplots)
        1    0.002    0.002    1.497    1.497 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/_subplots.py:45(make_subplots)
        1    0.000    0.000    1.170    1.170 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/graph_objs/_figure.py:736(update_layout)
        1    0.000    0.000    1.170    1.170 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:1378(update_layout)
        1    0.000    0.000    1.170    1.170 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:5096(update)
    802/2    0.009    0.000    0.626    0.313 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:3841(_perform_update)
        1    0.000    0.000    0.544    0.544 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/contextlib.py:139(__exit__)
        2    0.000    0.000    0.544    0.272 {built-in method builtins.next}
        2    0.000    0.000    0.544    0.272 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:2995(batch_update)
        1    0.000    0.000    0.544    0.544 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:2860(plotly_update)
     9697    0.072    0.000    0.443    0.000 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:51(_str_to_dict_path_full)
        1    0.000    0.000    0.417    0.417 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:2934(_perform_plotly_update)
        1    0.004    0.004    0.417    0.417 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:2611(_perform_plotly_relayout)
     1600    0.003    0.000    0.371    0.000 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:5842(__setitem__)
     1596    0.004    0.000    0.355    0.000 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:5726(_set_subplotid_prop)
     1598    0.010    0.000    0.353    0.000 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:5237(_set_compound_prop)
   103626    0.042    0.000    0.347    0.000 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:1811(_str_to_dict_path)
19302/14494    0.066    0.000    0.343    0.000 /home/bbm/miniforge3/envs/refl1d_py310/lib/python3.10/site-packages/plotly/basedatatypes.py:4659(__getitem__)


<pstats.Stats at 0x7ff048631750>

Guide de contribution

Ouvrir le guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez dans packages/python/plotly/plotly/basedatatypes.py, au niveau de BaseDatatype._perform_update, _check_path_in_prop_tree et _raise_on_invalid_property_error. Reproduisez le problème avec plotly.subplots.make_subplots(rows=20, cols=20) et comparez le profiling avec et sans le lookup. La solution finale doit fournir un moyen pris en charge d’éviter les similarity lookups inutiles pour les noms manquants tout en préservant la validation normale des properties.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
performance
Type d'issue
Fonctionnalité
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
45/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.