alibaba / alibaba/feathub

Supports over window on both limited and timed window in SparkProcessor

Open
#231 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Python
Stars
350
Forks
60
PR merge metrics
No merged PRs in 30d

Description

over_window_utils.py:

```python
def _get_spark_window_spec(
over_window_descriptor: "OverWindowDescriptor",
) -> WindowSpec:
if (
over_window_descriptor.limit is not None
and over_window_descriptor.window_size is not None
):
# TODO Supports aggregations on both limited and timed window
raise FeathubException(
"You cannot set window_size and limit of over window at the same time."
)
```

test_spark_processor.py

```python
# TODO: Add back following test cases after SparkProcessor supports aggregations
# on both limited and timed window
def test_over_window_on_join_field(self):
pass
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Read over_window_utils.py, especially _get_spark_window_spec, then inspect test_spark_processor.py and its TODO test_over_window_on_join_field case. Run the relevant SparkProcessor tests; done means aggregations can use both limited and timed windows, with coverage for the join-field case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, spark
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.