May you speed plotnine for big data (such as millions rows)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.8k
- Forks
- 254
- Avg merge
- 10h 35m
- Merged PRs (30d)
- 9
Description
When I plot a dataframe contain millions rows. The speed of ggplot is too slower than matplotlib.
Such as:
from plotnine import *
import numpy as np
import pandas as pd
When I use it in jupterlab, and run this code, the time is CPU times: user 8.3 s, sys: 212 ms, total: 8.51 s
Wall time: 2.34 s
%%time
fig, ax = plt.subplots()
ax.scatter(df.x, df.y)
plt.show()
But, when I use ggplot, the time is CPU times: user 3min 7s, sys: 6.82 s, total: 3min 14s
Wall time: 1min 13s
%%time
print(ggplot(df, aes('x', 'y')) + geom_point())
I check the run time of each step, and find the draw_unit of geom_point is too slow. It generate new column of size, stroke, color and fill as these codes although I just want to use same parameters for all points:
size = ((data['size']+data['stroke'])**2)*np.pi
stroke = data['stroke'] * SIZE_FACTOR
color = to_rgba(data['color'], data['alpha'])
if all(c is None for c in data['fill']):
fill = color
else:
fill = to_rgba(data['fill'], data['alpha'])
ax.scatter(x=data['x'],
y=data['y'],
s=size,
facecolor=fill,
edgecolor=color,
linewidth=stroke,
marker=data.loc[0, 'shape'],
zorder=params['zorder'])
The two to_rgba step is too slow, use about 20*2 seconds.
And the plt.show() cost 20 seconds.
If I directly provide plot parameters as scalar value to ax.scatter. The total run time of ggplot is :
CPU times: user 1min 36s, sys: 3.1 s, total: 1min 39s
Wall time: 11.3 s
Besides, layout.train_position and layout.map_position are also a little slow.
Whether can you speed the plotnine by generating a parameter dict for each data group? And thus you can didn't need to add plot parameter in self.data, but directly provide scalar values to plot function. Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the supplied JupyterLab benchmarks and profiling geom_point's draw_unit, especially the two to_rgba calls and parameter-column creation. Also inspect layout.train_position and layout.map_position as reported secondary costs. Done means preserving plot output while substantially reducing runtime for million-row data, verified against the example timings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- matplotlib, numpy, pandas, python
- Domain
- data-visualization, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100