has2k1 / has2k1/plotnine

May you speed plotnine for big data (such as millions rows)

Open
#482 0 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement Performance
Dominant language
Python
Stars
4.8k
Forks
254
Avg merge
10h 35m
Merged PRs (30d)
9

Description

When I plot a dataframe contain millions rows. The speed of ggplot is too slower than matplotlib.

Such as:

from plotnine import *
import numpy as np
import pandas as pd

When I use it in jupterlab, and run this code, the time is CPU times: user 8.3 s, sys: 212 ms, total: 8.51 s
Wall time: 2.34 s

%%time
fig, ax = plt.subplots()
ax.scatter(df.x, df.y)
plt.show()

But, when I use ggplot, the time is CPU times: user 3min 7s, sys: 6.82 s, total: 3min 14s
Wall time: 1min 13s

%%time
print(ggplot(df, aes('x', 'y')) + geom_point())

I check the run time of each step, and find the draw_unit of geom_point is too slow. It generate new column of size, stroke, color and fill as these codes although I just want to use same parameters for all points:

size = ((data['size']+data['stroke'])**2)*np.pi
stroke = data['stroke'] * SIZE_FACTOR
color = to_rgba(data['color'], data['alpha'])
if all(c is None for c in data['fill']):
            fill = color
else:
            fill = to_rgba(data['fill'], data['alpha'])

ax.scatter(x=data['x'],
                   y=data['y'],
                   s=size,
                   facecolor=fill,
                   edgecolor=color,
                   linewidth=stroke,
                   marker=data.loc[0, 'shape'],
                   zorder=params['zorder'])

The two to_rgba step is too slow, use about 20*2 seconds.
And the plt.show() cost 20 seconds.

If I directly provide plot parameters as scalar value to ax.scatter. The total run time of ggplot is :

CPU times: user 1min 36s, sys: 3.1 s, total: 1min 39s
Wall time: 11.3 s

Besides, layout.train_position and layout.map_position are also a little slow.

Whether can you speed the plotnine by generating a parameter dict for each data group? And thus you can didn't need to add plot parameter in self.data, but directly provide scalar values to plot function. Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the supplied JupyterLab benchmarks and profiling geom_point's draw_unit, especially the two to_rgba calls and parameter-column creation. Also inspect layout.train_position and layout.map_position as reported secondary costs. Done means preserving plot output while substantially reducing runtime for million-row data, verified against the example timings.

Written by the indexing model from the issue text.

Assessment

Tech stack
matplotlib, numpy, pandas, python
Domain
data-visualization, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.