HangfireIO / HangfireIO/Hangfire

DI error when deleting scheduled jobs from dashboard after upgrade

Open
#1,863 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.1k
Forks
1.8k
Avg merge
1h 19m
Merged PRs (30d)
1

Description

After I upgraded one of the minor versions (1.7.18 to 1.7.20), I'm getting an error when trying to delete jobs for retry. It's also happening if I try deleting the scheduled jobs. There is one caveat though - the first time during the application lifetime, the delete is successful. Every other time, it fails.

One thing to mention is that I'm not using standard DI for .NET 5 - I'm using DryIoc which is visible from the logs as well.

The error in the logs is the following:
```
at DryIoc.Throw.It(Int32 error, Object arg0, Object arg1, Object arg2, Object arg3) in /_/src/DryIoc/Container.cs:line 13568
at DryIoc.Container.ThrowIfContainerDisposed() in /_/src/DryIoc/Container.cs:line 684
at DryIoc.Container.ResolveAndCache(Int32 serviceTypeHash, Type serviceType, IfUnresolved ifUnresolved) in /_/src/DryIoc/Container.cs:line 418
at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in /_/src/DryIoc/Container.cs:line 356
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
at Hangfire.HangfireServiceCollectionExtensions.GetInternalServices(IServiceProvider provider, IBackgroundJobFactory& factory, IBackgroundJobStateChanger& stateChanger, IBackgroundJobPerformer& performer)
at Hangfire.DefaultClientManagerFactory.GetClient(JobStorage storage)
at Hangfire.Dashboard.AspNetCoreDashboardContext.GetBackgroundJobClient()
at Hangfire.Dashboard.RouteCollectionExtensions.<>c__DisplayClass3_0.b__0(DashboardContext context, String jobId)
at Hangfire.Dashboard.BatchCommandDispatcher.Dispatch(DashboardContext context)
at Hangfire.Dashboard.AspNetCoreDashboardMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)",
"message": "code: Error.ContainerIsDisposed;
message: Container is disposed and should not be used: \"container with ambient DryIoc.AsyncExecutionFlowScopeContext with scope {Name=null}
with Rules with {TrackingDisposableTransients, ResolveIEnumerableAsLazyEnumerable, CaptureContainerDisposeStackTrace, IgnoringReuseForFuncWithArgs} and without {ThrowOnRegisteringDisposableTransient, VariantGenericTypesInResolvedCollection}
with DefaultReuse=Singleton {Lifespan=1000}
with FactorySelector=SelectLastRegisteredFactory
with Made={FactoryMethod=ConstructorWithResolvableArguments, PropertiesAndFields=, ParameterSelector=} has been DISPOSED!
Dispose stack-trace at DryIoc.Container.Dispose()
at DryIoc.Microsoft.DependencyInjection.DryIocServiceScope.Dispose()
at Microsoft.AspNetCore.Http.Features.RequestServicesFeature.DisposeAsync()
at Microsoft.AspNetCore.Http.HttpResponse.<>c.<.cctor>b__36_2(Object disposable)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.FireOnCompleted()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.FireOnCompleted()
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.HandleRequest()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.HandleRequest()
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
```

I traced the error to the following commits - https://github.com/HangfireIO/Hangfire/commit/293bc41c3715fd48cd6b377f1369a79b2dcbf75f and https://github.com/HangfireIO/Hangfire/commit/07627d4261852381fc980906f48d32af2c281f1e

What I presume is happening is that `serviceProvider` that is injected in the `DefaultClientManagerFactory` will be OK the first time I try to delete one of the jobs. However, it gets disposed after the request is completed. The next time I try to delete the job, the new request is now trying to use the old, disposed `ServiceProvider` and is throwing this error.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing commits 293bc41c3715fd48cd6b377f1369a79b2dcbf75f and 07627d4261852381fc980906f48d32af2c281f1e, then inspect DefaultClientManagerFactory and the dashboard delete flow described in the stack trace. Reproduce repeated retry or scheduled-job deletions with DryIoc after upgrading. Done means later requests no longer attempt to resolve services from the disposed provider.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.