Need help understanding why function with event hub trigger periodically reprocesses data

We are in the process of expanding our Azure Function usage and lately I have seen an issue where our Event Hub triggered function seems to reprocess messages. I have not encountered this with any previous testing but now I am starting to see it quite frequently.

For reference, I am getting large spikes of outgoing messages when I view the metrics for the Event Hub:

Between 3:59 and 4:00, the outgoing message rate spikes up.

Looking at App Insights leading up to this time I see some exceptions which I believe are related to scaling and are normal:

- System.ArgumentOutOfRangeException: Ignoring out of date checkpoint with offset /sequence number because current persisted checkpoint has higher offset /sequence number (Parameter 'offset/sequenceNumber')
- Microsoft.WindowsAzure.Storage.StorageException: The lease ID specified did not match the lease ID for the blob. The lease ID specified did not match the lease ID for the blob.
- Microsoft.WindowsAzure.Storage.StorageException: There is already a lease present.

For some time after 4:00 I see other exceptions related to function timeout and out of memory, but I believe those are the result of my function having to reprocess a large amount of data.

When looking at the traces in App Insights, I see some of the below message leading up to 4:00:

There is only an error message and no exception/stack trace.

#### Investigative information

Please provide the following:

(timestamp and invocation id are not for a particular failure, but rather just an example of an execution around the time I started to see problems)

- Timestamp: 2021-03-15T20:57:27.1044292Z
- Function App version: 3.0.15405.0
- Invocation ID: b721d37c-a252-4cdb-a809-f8cda44a922f
- Region: centralus

#### Repro steps

N/A

#### Expected behavior

Function does not reprocess large amounts of data.

#### Actual behavior

Function appears to have reprocessed data.

#### Known workarounds

N/A

#### Related information

Provide any related information

- Using Custom Handlers

Start with the App Insights exceptions and traces around 2021-03-15T20:57:27Z, focusing on the Event Hub trigger, checkpoint and lease messages, scaling, timeouts, and out-of-memory errors. Review the custom handler context and available invocation data; done means identifying a reproducible cause and a verified way to prevent the apparent large-scale reprocessing.

Written by the indexing model from the issue text.