GoogleCloudPlatform / GoogleCloudPlatform/functions-framework-nodejs
Provide a mechanism to retry event processing without logging an error
- Ngôn ngữ chính
- TypeScript
- Star
- 1.4k
- Fork
- 181
- Chỉ số merge pull request
- Không có pull request nào được merge trong 30 ngày
Mô tả
Although Cloud Functions do allow a [retry mechanism](https://cloud.google.com/functions/docs/bestpractices/retries) when processing events, this mechanism is currently not very flexible. Indeed, in order for the processing to be retried, the function must throw an error.
Because the functions framework does not know the nature of the thrown error (Is it an uncaught error in a dependency? Is it an error thrown on purpose?), it will always log it as such along with its stack trace. While this makes sense in the general case, it also has effects on other Google services:
- The [Error Reporting](https://cloud.google.com/error-reporting/docs/) service will automatically pick up the error. If notifications have been enabled, this will also send a message to a different system (e.g. Slack, mail, etc).
- The logged error also counts towards the `logging.googleapis.com/log_entry_count` metric under the `ERROR` severity. This metric could be used in dashboard and alert policies. If the error is thrown intentionally only to retry processing, this adds noise to the metric and can possibly trigger false alerts.
There are many valid reasons why one would want to retry processing without explicitly logging an error. Actually, I would argue that the [main use case](https://cloud.google.com/functions/docs/bestpractices/retries#use_retry_to_handle_transient_errors) detailed in the documentation (transient errors) should probably not log an error. If an error is transient (e.g. a timeout or a network error when calling an API) and is expected to occur from time to time, there's not much value in counting it as a generic error in Error Reporting and Cloud Logging / Monitoring. In fact, those transient errors can probably be reported more precisely using Cloud Logging and Monitoring directly (not relying on the `functions-framework` wrapper), along with other statistics about the API calls (e.g. latency, endpoint name, etc).
Conversely, I'd argue that we want an error to be logged and picked us as such by downstream GCP services when it is unexpected. Moreover if the error is unexpected, it should probably not be retried because it could be a bug in the code (as the GCP documentation indeed calls out).
**✨ Ideal behaviour**
Summing up the previous points, the ideal behaviour for me would be to:
- Log all (uncaught) errors as errors, but never retry event processing (return status code `204`). This corresponds to unforeseen errors that might be bugs.
- Provide an explicit way to ask for a later retry of the processing. For example, transient errors can explicitly be caught in user code, which can then return or throw a special value to indicate the need to retry processing. (In this case, the function would return status code `500`.)
This would reduce the risk of setting up a Cloud Function with the `retry` flag and ending up with endless retries. The [solutions](https://cloud.google.com/functions/docs/bestpractices/retries#set_an_end_condition_to_avoid_infinite_retry_loops) provided by the documentation involve custom logic and are in my opinion prone to errors and/or suboptimal.
**🛠️ Current workaround**
My current workaround is actually a wrapper around all my Cloud Functions which:
- Catches all uncaught errors of "unknown" types, and logs them as errors. Those errors are not rethrown by the wrapper and the execution terminates successfully. This is completely transparent to the `functions-frameworks`, which thinks the processing ended successfully. Processing is not retried. Logged errors do appear in Error Reporting / Cloud Monitoring error metrics, as they need to be reviewed and possibly fixed by developers.
- Catches errors of a special type (`RetryableError`) and rethrows them. Only an `INFO`-level log is written because the `RetryableError` marks an explicit need in user code to retry processing. Because a rejection is passed to the `functions-framework`, the error is logged and processing is retried.
**Obvisouly, this workaround does not fully eliminate the problem** because retryable errors are still logged as errors by the `functions-framework`. However it does help to:
- Avoid retrying processing in case the error is an unexpected bug.
- Make it explicit which errors should be retried, both for code readability and in tests.
**💡 Breaking change-free suggestion**
The ideal behaviour described above would result in a huge breaking change and the idea would probably be quickly dismissed. However I think there is a smaller, non-breaking, change that can bring more flexibility to developers: simply provide an explicit value or error that can be returned/thrown by the function, which does not involve logging an error but still causes the `functions-framework` to return a `500` status code. This would be a small change to [sendResponse](https://github.com/GoogleCloudPlatform/functions-framework-nodejs/blob/978054b786ce180e350b8484b65e0ea116abba1f/src/invoker.ts#L42), checking that `result` or `err` is of a specific type (or whatever check you feel most comfortable with).
This solution does solve the main problem which is that processing cannot currently be retried without logging an error. Because uncaught errors would still be retried (and also logged as errors), this does not alleviate the risk of an event being endlessly retried in case of a bug in the function's code. However this is clearly [documented](https://cloud.google.com/functions/docs/bestpractices/retries#why_event-driven_functions_fail_to_complete) and can be solved in user code (e.g. like it is in my current workaround).
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu trong src/invoker.ts tại sendResponse, nơi issue xác định là điểm tích hợp để phân biệt tín hiệu retry rõ ràng với các lỗi thông thường. Xem xét cách result và err hiện đang quyết định hành vi phản hồi và logging. Hoàn thành khi một giá trị hoặc lỗi rõ ràng có thể tạo ra HTTP 500 và kích hoạt retry sự kiện mà không logging lỗi, trong khi các lỗi thông thường chưa được bắt vẫn giữ nguyên hành vi hiện tại.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- google-cloud, nodejs, typescript
- Lĩnh vực
- api, backend, cloud
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 35/100