Azure / Azure/azure-functions-sql-extension

Keep track of rows that were ignored because they hit the max attempt limit

Open
#1,106 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
130
Forks
71
Avg merge
4d 8h
Merged PRs (30d)
4

Description

Currently if a row fails to be processed 5 times it will be ignored for all future attempts. Logging in https://github.com/Azure/azure-functions-sql-extension/pull/1103 is being added to tell when this happens, but it would be even better to have some way of knowing exactly what rows were ignored (we can't log that information since it contains customer content).

The lease table contains this information (row keys and attempt count), but currently this is cleared once we update the last sync version in the BuildUpdateTablesPostInvocation method.

This is done so that while that version of the row is ignored forever, we will still try to process further changes in the row (in case whatever issue that caused the original failures was fixed).

But doing this means it's difficult to know what rows cause problems and easily track/reprocess those rows. Keeping a historical record of rows that failed to be processed can be hugely beneficial for auditing purposes - similar to the [Dead letter queue](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues) that the service bus binding implements.

Contributor guide

Open the contributing guide

Research direction

Start at BuildUpdateTablesPostInvocation and inspect how the lease table's row keys and attempt counts are cleared after the last sync version is updated. Read the logging context in PR 1103, then define how a historical record should remain available for auditing and reprocessing. Done means rows that hit the maximum attempt limit and their attempt counts remain trackable after synchronization.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, sql
Domain
backend, database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.