elsa-workflows / elsa-workflows/elsa-core
Elsa.EntityFrameworkCore.Store does not respect updated credentials via `reloadOnChange: true`
- Dominant language
- C#
- Stars
- 7.9k
- Forks
- 1.5k
- Avg merge
- 15h 22m
- Merged PRs (30d)
- 114
Description
## Description
In enterprise environments, it is common to rotate credentials or URLs for databases and other services via updating configuration files on the system, such as via [Hashicorp Vault's sidecar vault-agent](https://developer.hashicorp.com/vault/tutorials/app-integration/dotnet-vault-agent). However, due to the way Recurring Tasks are constructed, updates to the system configuration are not discovered.
In addition, if a failure happens in a recurring task (such as an authentication failure for expired credentials), exceptions are not caught, causing the full application to terminate with an unhandled exception.
## Steps to Reproduce
To reproduce the crash:
1. Set up Elsa using Entity Framework.
2. Run the application.
3. Change the password to your database.
4. Wait for the `DefaultBookmarkQueuePurger` to run.
**Actual result:** The application crashes the next time the `DefaultBookmarkQueuePurger` attempts to run.
**Expected result:** Error-level log message, but the application should not crash (or, crash if an exception handler is not added)
To reproduce the full correct behavior:
1. Add a JSON file to your configuration builder via `.AddJsonFile("/absolute/file/path", reloadOnChange: true)`.
2. Set up Elsa using Entity Framework based on the added file.
3. Run the application.
4. Ensure Elsa is running correctly.
5. Change the password to your database and update the corresponding JSON file.
6. Wait for the `DefaultBookmarkQueuePurger` to run.
**Actual result:** The application crashes the next time the `DefaultBookmarkQueuePurger` attempts to run.
**Expected result:** The `DefaultBookmarkQueuePurger` continues to run successfully.
## Environment
- **Elsa Package Version**: Elsa 3.3.1, Elsa.EntityFrameworkCore.PostgreSql 3.3.1
- **Operating System**: Docker via `mcr.microsoft.com/dotnet/aspnet:9.0`
- **Browser and Version**: N/A
## Log Output
The stack trace received when this occurs:
```
Unhandled exception. Npgsql.PostgresException (0x80004005): 42501: permission denied for schema (redacted)
POSITION: 195
at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(CommandBehavior behavior, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.InitializeReaderAsync(AsyncEnumerator enumerator, CancellationToken cancellationToken)
at Npgsql.EntityFrameworkCore.PostgreSQL.Storage.Internal.NpgsqlExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.AsyncEnumerator.MoveNextAsync()
at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
at Elsa.EntityFrameworkCore.Store`2.QueryAsync(Func`2 query, Func`4 onLoading, Boolean ignoreQueryFilters, CancellationToken cancellationToken)
at Elsa.EntityFrameworkCore.Store`2.QueryAsync(Func`2 query, Func`4 onLoading, Boolean ignoreQueryFilters, CancellationToken cancellationToken)
at Elsa.EntityFrameworkCore.Store`2.QueryAsync(Func`2 query, CancellationToken cancellationToken)
at Open.Linq.AsyncExtensions.Extensions.LongCount[TSource](Task`1 source)
at Elsa.EntityFrameworkCore.Modules.Runtime.EFBookmarkQueueStore.PageAsync[TOrderBy](PageArgs pageArgs, BookmarkQueueFilter filter, BookmarkQueueItemOrder`1 orderBy, CancellationToken cancellationToken)
at Elsa.Workflows.Runtime.DefaultBookmarkQueuePurger.PurgeAsync(CancellationToken cancellationToken)
at Elsa.Common.Multitenancy.TaskExecutor.ExecuteInternalAsync(ITask task, Func`1 action, CancellationToken cancellationToken)
at Elsa.Common.Multitenancy.TaskExecutor.ExecuteTaskAsync(ITask task, CancellationToken cancellationToken)
at Elsa.Common.Multitenancy.EventHandlers.StartRecurringTasks.<>c__DisplayClass5_1.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Elsa.Common.RecurringTasks.ScheduledTimer.Callback(Object state)
at System.Threading.Tasks.Task.<>c.b__128_1(Object state)
at System.Threading.QueueUserWorkItemCallback.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
Exception data:
Severity: ERROR
SqlState: 42501
MessageText: permission denied for schema (redacted)
Position: 195
File: aclchk.c
Line: 3655
Routine: aclcheck_error
```
## Troubleshooting Attempts
- Verified configuration is reloading properly by using `IOptionsMonitor` as well as `IConfiguration`. (In Kubernetes, it is unclear if `ENV DOTNET_USE_POLLING_FILE_WATCHER=true` is required, but we added it. This is reproducible outside of Kubernetes as described above.)
## Workaround
We were able to work around the issue by using the following implementation:
```csharp
using Elsa.Workflows.Runtime;
using Microsoft.Extensions.DependencyInjection;
namespace Redacted.Data.Workflows;
internal class CustomBookmarkQueuePurger : IBookmarkQueuePurger
{
private readonly IServiceScopeFactory serviceScopeFactory;
public CustomBookmarkQueuePurger(IServiceScopeFactory serviceScopeFactory)
{
this.serviceScopeFactory = serviceScopeFactory;
}
public async Task PurgeAsync(CancellationToken cancellationToken = default)
{
using var scope = serviceScopeFactory.CreateScope();
var bookmarkQueuePurger = ActivatorUtilities.GetServiceOrCreateInstance(scope.ServiceProvider);
await bookmarkQueuePurger.PurgeAsync(cancellationToken);
}
}
```
and registering it with:
```csharp
services.AddScoped();
```
## Additional Context
If there's additional context I can provide, I'm willing to update it.
## Related Issues
I could not find related issues.
Contributor guide
Assessment
This issue has not been assessed yet.