Incorrect order of message ack message processing
- Dominant language
- Scala
- Stars
- 6.8k
- Forks
- 1.2k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 2
Description
This may be a duplicate of #4134
While experimenting local with disabling activation polling (#4088) I noticed that the first blocking activation at controller received a 202 instead of 200 response. This is because while `ResultMessage` is sent from invoker and received at controller first, the controller executes `processCompletion()` (with the `CompletionMessage`) before executing `processResult()`.
Subsequent activations are processed in the expected order (`processResult()` then `processCompletion()`)
## Environment details:
* Mac OS
## Steps to reproduce the issue:
1. run controller + invoker locally
2. send a blocking activation
3.
## Provide the expected results and outputs:
Expected logs to indicate in this order:
```
...
[ShardingContainerPoolBalancer] received result ack for
...
[ShardingContainerPoolBalancer] received completion ack for
...
```
## Provide the actual results and outputs:
Actual order for *first* activation at controller is opposite:
```
...
[ShardingContainerPoolBalancer] received completion ack for
...
[ShardingContainerPoolBalancer] received result ack for
...
```
## Additional information you deem important:
I think this is related to `ShardingContainerPoolBalancer.processAcknowledgement()`, which does:
```
Future {
val raw = new String(bytes, StandardCharsets.UTF_8)
AcknowledegmentMessage.parse(raw) match {
}
}
```
Testing locally, moving the `Future{}` to after the `AcknowledegmentMessage.parse(raw)` fixes the problem.
Contributor guide
Research direction
Start in ShardingContainerPoolBalancer.processAcknowledgement() and inspect where AcknowledegmentMessage.parse(raw) is called relative to the Future. Reproduce with a local controller and invoker while sending a blocking activation, then compare the acknowledgement logs. Done means the first activation receives and processes the result acknowledgement before the completion acknowledgement and returns the expected 200 response.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 38/100