Supports over window on both limited and timed window in SparkProcessor
- Dominant language
- Python
- Stars
- 350
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
over_window_utils.py:
```python
def _get_spark_window_spec(
over_window_descriptor: "OverWindowDescriptor",
) -> WindowSpec:
if (
over_window_descriptor.limit is not None
and over_window_descriptor.window_size is not None
):
# TODO Supports aggregations on both limited and timed window
raise FeathubException(
"You cannot set window_size and limit of over window at the same time."
)
```
test_spark_processor.py
```python
# TODO: Add back following test cases after SparkProcessor supports aggregations
# on both limited and timed window
def test_over_window_on_join_field(self):
pass
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Read over_window_utils.py, especially _get_spark_window_spec, then inspect test_spark_processor.py and its TODO test_over_window_on_join_field case. Run the relevant SparkProcessor tests; done means aggregations can use both limited and timed windows, with coverage for the join-field case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100