apache / apache/openwhisk

Incorrect order of message ack message processing

Open
#4,146 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
6.8k
Forks
1.2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

This may be a duplicate of #4134
While experimenting local with disabling activation polling (#4088) I noticed that the first blocking activation at controller received a 202 instead of 200 response. This is because while `ResultMessage` is sent from invoker and received at controller first, the controller executes `processCompletion()` (with the `CompletionMessage`) before executing `processResult()`.

Subsequent activations are processed in the expected order (`processResult()` then `processCompletion()`)

## Environment details:

* Mac OS
## Steps to reproduce the issue:

1. run controller + invoker locally
2. send a blocking activation
3.

## Provide the expected results and outputs:

Expected logs to indicate in this order:
```
...
[ShardingContainerPoolBalancer] received result ack for
...
[ShardingContainerPoolBalancer] received completion ack for
...
```

## Provide the actual results and outputs:
Actual order for *first* activation at controller is opposite:
```
...
[ShardingContainerPoolBalancer] received completion ack for
...
[ShardingContainerPoolBalancer] received result ack for
...
```
## Additional information you deem important:
I think this is related to `ShardingContainerPoolBalancer.processAcknowledgement()`, which does:

```
Future {
val raw = new String(bytes, StandardCharsets.UTF_8)
AcknowledegmentMessage.parse(raw) match {

}
}
```

Testing locally, moving the `Future{}` to after the `AcknowledegmentMessage.parse(raw)` fixes the problem.

Contributor guide

Open the contributing guide

Research direction

Start in ShardingContainerPoolBalancer.processAcknowledgement() and inspect where AcknowledegmentMessage.parse(raw) is called relative to the Future. Reproduce with a local controller and invoker while sending a blocking activation, then compare the acknowledgement logs. Done means the first activation receives and processes the result acknowledgement before the completion acknowledgement and returns the expected 200 response.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.