HangfireIO / HangfireIO/Hangfire

Azure cloud database connection resiliency

Open
#2,477 1 comment 0 reactions 0 assignees View on GitHub
t: question
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

We are using Hangfire in a Linux docker container running in Azure. In Azure we experience quite often database connection issues. Example Exception:
A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 35 - An internal exception was caught) at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

We get these exception when accessing the database with our code and also when Hangfire tries to access the db.
After some research I came accross the following GitHub issues:
https://github.com/dotnet/SqlClient/issues/2103#issuecomment-1764206103
https://github.com/dotnet/SqlClient/issues/1773

There seems to be a bug in the Microsoft Linux version of the SQLClient which does not hand over the correct error number when creating the SQLException. The error code is always 0 in these resiliency cases.
The Windows version of the SQLClient does not have this bug and therefore EFCore can handle these resiliency issues and does a reconnect / retry the database operation. On Linux it can't because of the missing error number.

We were able to implement a custom execution strategy to handle these resiliency problems as shown here:
https://github.com/dotnet/SqlClient/issues/2103#issuecomment-1798070309

Have you ever been made aware of these resiliency issues under Linux or could you implement something similar in your code base?
The issue we currently face is that our code now does no longer create these exceptions but when Hangfire wants to access it's database it fails with the error message mentioned above.

So my question is if you could implement such a custom resiliency strategy in your code?

Thanks,
Christian

Contributor guide

Open the contributing guide

Research direction

Start by tracing Hangfire's database access and reviewing the Microsoft.Data.SqlClient issues linked in the report, especially the custom execution strategy example. Determine where Linux transport errors are classified and define completion as Hangfire retrying or reconnecting for the reported Azure database failures without breaking existing database behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, docker, linux, sql
Domain
backend, cloud, database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.