apache / apache/wayang

check for the case where the index matter

Open
#221 0 comments 0 reactions 0 assignees View on GitHub
todo
Dominant language
Java
Stars
274
Forks
140
Avg merge
5d 16h
Merged PRs (30d)
4

Description

check for the case where the index matter

https://github.com/apache/incubator-wayang/blob/77a5d3b5854f59b9cad1f9e9921b849ccc5cca58/python/src/pywy/platforms/jvm/operator/jvm_unary_filter.py#L61

```python

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

from typing import Set, List, Type

from pywy.core.channel import CH_T, ChannelDescriptor
from pywy.operators.unary import FilterOperator
from pywy.platforms.jvm.channels import DISPATCHABLE_CHANNEL_DESCRIPTOR, DispatchableChannel
from pywy.platforms.jvm.operator.jvm_execution_operator import JVMExecutionOperator
from pywy.platforms.commons.channels import (
CommonsCallableChannel
)
from pywy.platforms.jvm.serializable.wayang_jvm_operator import WayangJVMMappartitionOperator, WayangJVMOperator

class JVMFilterOperator(FilterOperator, JVMExecutionOperator):

def __init__(self, origin: FilterOperator = None, **kwargs):
predicate = None if origin is None else origin.predicate
super().__init__(predicate)
self.set_context(**kwargs)

def execute(self, inputs: List[Type[CH_T]], outputs: List[Type[CH_T]]):
self.validate_channels(inputs, outputs)
udf = self.predicate
if isinstance(inputs[0], DispatchableChannel):
py_in_dispatch_channel: DispatchableChannel = inputs[0]
py_out_dispatch_channel: DispatchableChannel = outputs[0]

def func(iterator):
return filter(udf, iterator)

py_out_dispatch_channel.accept_callable(
CommonsCallableChannel.concatenate(
func,
py_in_dispatch_channel.provide_callable()
)
)

op: WayangJVMOperator = py_in_dispatch_channel.provide_dispatchable()

if isinstance(op, WayangJVMMappartitionOperator):
py_out_dispatch_channel.accept_dispatchable(op)
return

current: WayangJVMMappartitionOperator = WayangJVMMappartitionOperator(self.name)
# TODO check for the case where the index matter
op.connect_to(0, current, 0)
self.close_operator(op)
py_out_dispatch_channel.accept_dispatchable(current)

else:
raise Exception("Channel Type does not supported")

def get_input_channeldescriptors(self) -> Set[ChannelDescriptor]:
return {DISPATCHABLE_CHANNEL_DESCRIPTOR}

def get_output_channeldescriptors(self) -> Set[ChannelDescriptor]:
return {DISPATCHABLE_CHANNEL_DESCRIPTOR}

```

456c3f1256e226291d2b657267900dc092281993

Contributor guide

Open the contributing guide

Research direction

Start in python/src/pywy/platforms/jvm/operator/jvm_unary_filter.py at the TODO near op.connect_to(0, current, 0), then trace WayangJVMMappartitionOperator and connect_to usage to determine when the input index matters. Reproduce or inspect an affected filter execution and define the expected dispatchable connection behavior before adding coverage; done means the relevant indexed case is handled without breaking existing JVM filtering.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.