apache / apache/beam

TFX pipelines experience a huge blowup in intermediate data size

Open
#19,342 0 comments 0 reactions 0 assignees View on GitHub
bug flink P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The elements in TFX intermediate collections are dictionaries of (typically single-element) numpy arrays, which are (relatively) expensive to serialize (e.g. using pickle for the numpy wrapper of a primitive int/float, repeating the column names in every element).

Though it'd be good to use a better intermediate representation, this is exacerbated because the fusion algorithm does not pack as much possible into executable stages.

Imported from Jira [BEAM-6243](https://issues.apache.org/jira/browse/BEAM-6243). Original Jira may contain additional context.
Reported by: robertwb.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.