[Proposal] Internal retry and DLQ mechanism

We're using OpenWhisk on IBM Cloud for streaming ETL and have implemented a custom retry mechanism as an ad-hoc addition to our pipeline. However, there are still two scenarios where lack of internal retry means we are out of options: internal failures and network failure at the machine running OpenWhisk.

If for whatever reason OpenWhisk cannot connect to any outside system, it becomes a black box that can only be investigated later by referring to log outputs. However, for security and practical reasons, using logs is not a feasible option for dealing with a high throughput system. Furthermore, log outputs might not be available for internal failures at all e.g. OOM crashes

The only alternative is to validate the information from upstream services that pass data to OpenWhisk (in our case Kafka) to downstream services; e.g. having an external arbiter that checks to see if there is a successful event in the downstream services for every message in our upstream service. However, that is not practically feasible because downstream services can be e.g. SaaS products that are not controlled by us and do not expose the required information.

My proposal is for OpenWhisk to handle retry internally after a payload has been handed off to it and it has acknowledged receiving it.

Thanks