significant speedup for "to_scalar_or_list"
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 18.8k
- Forks
- 2.8k
- Avg merge
- 16h 26m
- Merged PRs (30d)
- 21
Description
hello,
While investigating a slowness in plotly, I have stumbled upon the to_scalar_or_list function (https://github.com/plotly/plotly.py/blob/abd86092e048d5c8b02da65123824b75e4311838/packages/python/plotly/_plotly_utils/basevalidators.py#L30) that was taking much time.
After some tinkering, I came with the two following changes that vastly improves the performance:
-
move out of the function the lines 38/39 (https://github.com/plotly/plotly.py/blob/abd86092e048d5c8b02da65123824b75e4311838/packages/python/plotly/_plotly_utils/basevalidators.py#L38) with the
get_moduleas it is slow and run each time the function is called (when handling a list of 10k elements, 10k calls) ==> can this be done once in plotly instead of dynamically in each function ? (I see the get_module is also used in many other places in the package) -
move the simplest case (v is a basic type) first as for the case of an iterable of size N, it will first do lot of complex tests for the iterable and then N times also all the complex tests for each items.
So at the end, it looks like
np = get_module("numpy", should_load=False)
pd = get_module("pandas", should_load=False)
# Utility functions
# -----------------
def to_scalar_or_list(v):
# Handle the case where 'v' is a non-native scalar-like type,
# such as numpy.float32. Without this case, the object might be
# considered numpy-convertable and therefore promoted to a
# 0-dimensional array, but we instead want it converted to a
# Python native scalar type ('float' in the example above).
# We explicitly check if is has the 'item' method, which conventionally
# converts these types to native scalars.
# check first for the simple case
if isinstance(v,(int,float,str)):
return v
if np and np.isscalar(v) and hasattr(v, "item"):
return v.item()
if isinstance(v, (list, tuple)):
return [to_scalar_or_list(e) for e in v]
elif np and isinstance(v, np.ndarray):
if v.ndim == 0:
return v.item()
return [to_scalar_or_list(e) for e in v]
elif pd and isinstance(v, (pd.Series, pd.Index)):
return [to_scalar_or_list(e) for e in v]
elif is_numpy_convertable(v):
return to_scalar_or_list(np.array(v))
else:
return v
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/python/plotly/_plotly_utils/basevalidators.py at to_scalar_or_list and inspect the nearby get_module and is_numpy_convertable usage. Compare behavior and performance for native scalars, lists or tuples, NumPy arrays, and pandas Series or Index values. Done means the proposed cases retain their existing conversions while avoiding unnecessary repeated work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100