check for the case where the index matter
- Dominant language
- Java
- Stars
- 274
- Forks
- 140
- Avg merge
- 5d 16h
- Merged PRs (30d)
- 4
Description
check for the case where the index matter
https://github.com/apache/incubator-wayang/blob/77a5d3b5854f59b9cad1f9e9921b849ccc5cca58/python/src/pywy/platforms/jvm/operator/jvm_unary_filter.py#L61
```python
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from typing import Set, List, Type
from pywy.core.channel import CH_T, ChannelDescriptor
from pywy.operators.unary import FilterOperator
from pywy.platforms.jvm.channels import DISPATCHABLE_CHANNEL_DESCRIPTOR, DispatchableChannel
from pywy.platforms.jvm.operator.jvm_execution_operator import JVMExecutionOperator
from pywy.platforms.commons.channels import (
CommonsCallableChannel
)
from pywy.platforms.jvm.serializable.wayang_jvm_operator import WayangJVMMappartitionOperator, WayangJVMOperator
class JVMFilterOperator(FilterOperator, JVMExecutionOperator):
def __init__(self, origin: FilterOperator = None, **kwargs):
predicate = None if origin is None else origin.predicate
super().__init__(predicate)
self.set_context(**kwargs)
def execute(self, inputs: List[Type[CH_T]], outputs: List[Type[CH_T]]):
self.validate_channels(inputs, outputs)
udf = self.predicate
if isinstance(inputs[0], DispatchableChannel):
py_in_dispatch_channel: DispatchableChannel = inputs[0]
py_out_dispatch_channel: DispatchableChannel = outputs[0]
def func(iterator):
return filter(udf, iterator)
py_out_dispatch_channel.accept_callable(
CommonsCallableChannel.concatenate(
func,
py_in_dispatch_channel.provide_callable()
)
)
op: WayangJVMOperator = py_in_dispatch_channel.provide_dispatchable()
if isinstance(op, WayangJVMMappartitionOperator):
py_out_dispatch_channel.accept_dispatchable(op)
return
current: WayangJVMMappartitionOperator = WayangJVMMappartitionOperator(self.name)
# TODO check for the case where the index matter
op.connect_to(0, current, 0)
self.close_operator(op)
py_out_dispatch_channel.accept_dispatchable(current)
else:
raise Exception("Channel Type does not supported")
def get_input_channeldescriptors(self) -> Set[ChannelDescriptor]:
return {DISPATCHABLE_CHANNEL_DESCRIPTOR}
def get_output_channeldescriptors(self) -> Set[ChannelDescriptor]:
return {DISPATCHABLE_CHANNEL_DESCRIPTOR}
```
456c3f1256e226291d2b657267900dc092281993
Contributor guide
Research direction
Start in python/src/pywy/platforms/jvm/operator/jvm_unary_filter.py at the TODO near op.connect_to(0, current, 0), then trace WayangJVMMappartitionOperator and connect_to usage to determine when the input index matters. Reproduce or inspect an affected filter execution and define the expected dispatchable connection behavior before adding coverage; done means the relevant indexed case is handled without breaking existing JVM filtering.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100