Deduplicate Python UDF invocation
Open
type:bug
- Dominant language
- Python
- Stars
- 350
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
In test_sliding_window_transform._test_late_data, when the input dataframe contains 5 records, the UDF `sleep` would be invoked 15 times. Figure out why and fix this bug to reduce performance overhead.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in test_sliding_window_transform._test_late_data and reproduce the case with an input dataframe containing 5 records. Trace why the Python UDF sleep is invoked 15 times, then verify the fix by rerunning this test and confirming that redundant invocations and their performance overhead are reduced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100