apache / apache/beam

Memoize DataFrame operations

Open
#20,875 0 comments 0 reactions 0 assignees View on GitHub
core dataframe dsl improvement P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Currently performing an operation on a deferred dataframe always produces a _new_ deferred dataframe. This means a call like to_pcollection(df.mean(), df.mean()), will produce two distinct PCollections duplicating the same computation.

This is particularly problematic for the interactive use-case where, to_pcollection is used inside of ib.collect() in combination with PCollection caching. Collecting df.mean() two different times will duplicate the computation unnecessarily.

We should cache the output expressions produced by operations to prevent this.

We need to be mindful of inplace operations when implementing this:
- Two calls to df.mean() should produce the same result iff df has not been mutated in between.
- If the output of one call to df.mean() is mutated, it must not mutate the output of another call to df.mean().

Imported from Jira [BEAM-12245](https://issues.apache.org/jira/browse/BEAM-12245). Original Jira may contain additional context.
Reported by: bhulette.

Contributor guide

Open the contributing guide

Research direction

Start by tracing deferred DataFrame operations around df.mean(), then follow how to_pcollection() is used inside ib.collect(). Define how cached outputs behave across repeated calls and inplace mutations; done means equivalent unmutated operations reuse their output without allowing one returned result to mutate another.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.