apache / apache/openwhisk-package-kafka

Handle Action Error

Open
#271 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
30
Forks
40
PR merge metrics
No merged PRs in 30d

Description

Hello guys,
how can errors be handled when using messageHubFeed as a trigger for an openwhisk action?

Let's take the following example scenario:
TopicA contains the messages: M1, M2, M3, M4, M5

The openwhisk action Action1 is bound to a trigger for TopicA.

Action1 persists messages in Cloudant.

The trigger is sucessfully fired with {"messages": [M1, M2, M3]}.
Now assume that Cloudant is unavailable or the action crashes/fails.

As far as I know, the offset has already been commited, so these messages won't ever be redelivered/retried.
And maybe following trigger/action invocations (in case of cloudant being down for let's say 5 minutes) may end the same.

So to sum up: If the action fails, messageHubFeed ignores that and fires the trigger for the next messages. Whether they can be processed or not. In worst case all messages get delivered but never successfully processed.
In such a case it would be nice to pause the delivery until the action can process the messages again.

"Messages can't currently be processed, it is not good to deliver more of them, let's queue them up (kafka can do this) and try to continue delivery in 5 Minutes".

For sure I understand that a poisoned message should not halt the processing and may be skipped. But what can we do in such a "Database is down"-scenario?

Can/Should the processing be paused?

Do we need to monitor the activation records and manually resolve all the failed ones?

Should all affected messages be sent to a Dead-Letter-Queue/Topic?
And what if that fails, too (timeout, network partitioning, ...)?

Does anyone have some ideas or experience on how to deal with that kind of scenarios?

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reviewing the messageHubFeed trigger's offset handling and activation behavior for failed OpenWhisk actions, then compare the existing project guidance with the Cloudant outage scenario. Done would require a documented or implemented decision for retries, pausing delivery, poisoned messages, and dead-letter handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, python
Domain
cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.