[Task]: Only handle nulls for nullable columns in pandas to Beam row conversion
Open
dataframe
dsl
P2
task
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What needs to happen?
Currently when we convert a pandas DataFrame to Beam Rows we check for nulls in every column. This is wasteful, we should instead only do null-checking for the columns that we know to be nullable.
### Issue Priority
Priority: 2
### Issue Component
Component: dsl-dataframe
Contributor guide
Research direction
Start by locating the pandas DataFrame to Beam Row conversion in the dsl-dataframe component and inspect how nulls are checked for each column. The work is done when null-checking is limited to columns known to be nullable while conversion behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100