apache / apache/beam

[Task]: Only handle nulls for nullable columns in pandas to Beam row conversion

Open
#22,948 0 comments 0 reactions 0 assignees View on GitHub
dataframe dsl P2 task
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What needs to happen?

Currently when we convert a pandas DataFrame to Beam Rows we check for nulls in every column. This is wasteful, we should instead only do null-checking for the columns that we know to be nullable.

### Issue Priority

Priority: 2

### Issue Component

Component: dsl-dataframe

Contributor guide

Open the contributing guide

Research direction

Start by locating the pandas DataFrame to Beam Row conversion in the dsl-dataframe component and inspect how nulls are checked for each column. The work is done when null-checking is limited to columns known to be nullable while conversion behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.