elsa-workflows / elsa-workflows/elsa-core

Unable to access existing secrets after the server/container restart

Open
#6,716 2 comments 0 reactions 0 assignees View on GitHub
triaged
Dominant language
C#
Stars
7.9k
Forks
1.5k
Avg merge
15h 22m
Merged PRs (30d)
114

Description

## Description
We are encountering an issue with Elsa 3.4.0-rc1 running on Heroku, where encrypted secrets (managed via IDataProtectionProvider in DataProtectionEncryptor) fail to decrypt after a restart of the server.

Heroku automatically restarts dynos at least once every 24 hours, and its file system is ephemeral, which results in the loss of Data Protection keys.

## Steps to Reproduce

- Save the secrets from Elsa studio
- Deploy or restart the server
- Now try to access the existing secrets, its fails

Image

## Logs

> 2025-06-06T22:15:41.920156+00:00 heroku[router]: Request finished HTTP/1.1 GET http://test.herokuapp.com/elsa/api/secrets/a75ee2805ecc40b5/input - 500 - text/html;+charset=utf-8 4.5809ms
> 2025-06-06T22:15:41.920156+00:00 heroku[router]: at=info method=GET path="/elsa/api/secrets/a75ee2805ecc40b5/input" host=test.herokuapp.com request_id=451f50d0-920f-4a4e-2880-bcb8934773f6 fwd="24.98.168.93" dyno=web.1 connect=0ms service=4ms status=500 bytes=2695 protocol=http1.1 tls=true tls_version=unknown
> 2025-06-06T22:15:41.916484+00:00 app[web.1]: info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
> 2025-06-06T22:15:41.916512+00:00 app[web.1]: Request starting HTTP/1.1 GET http://test.herokuapp.com/elsa/api/secrets/a75ee2805ecc40b5/input - - -
> 2025-06-06T22:15:41.916809+00:00 app[web.1]: info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
> 2025-06-06T22:15:41.916809+00:00 app[web.1]: Executing endpoint 'HTTP: GET /elsa/api/secrets/{id}/input'
> 2025-06-06T22:15:41.918709+00:00 app[web.1]: info: Microsoft.EntityFrameworkCore.Database.Command[20101]
> 2025-06-06T22:15:41.918709+00:00 app[web.1]: Executed DbCommand (1ms) [Parameters=[@__Id_0='a75ee2805ecc40b5'], CommandType='Text', CommandTimeout='30']
> 2025-06-06T22:15:41.918710+00:00 app[web.1]: SELECT s."Id", s."CreatedAt", s."Description", s."EncryptedValue", s."ExpiresAt", s."ExpiresIn", s."IsLatest", s."LastAccessedAt", s."Name", s."Owner", s."Scope", s."SecretId", s."Status", s."TenantId", s."UpdatedAt", s."Version"
> 2025-06-06T22:15:41.918711+00:00 app[web.1]: FROM "Elsa"."Secrets" AS s
> 2025-06-06T22:15:41.918711+00:00 app[web.1]: WHERE s."TenantId" IS NULL AND s."Id" = @__Id_0
> 2025-06-06T22:15:41.918946+00:00 app[web.1]: info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
> 2025-06-06T22:15:41.918946+00:00 app[web.1]: Executed endpoint 'HTTP: GET /elsa/api/secrets/{id}/input'
> 2025-06-06T22:15:41.919353+00:00 app[web.1]: fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
> 2025-06-06T22:15:41.919354+00:00 app[web.1]: An unhandled exception has occurred while executing the request.
> 2025-06-06T22:15:41.919358+00:00 app[web.1]: System.Security.Cryptography.CryptographicException: The key {4774f754-b089-4e1e-b09c-c38613602db7} was not found in the key ring. For more information go to https://aka.ms/aspnet/dataprotectionwarning
> 2025-06-06T22:15:41.919358+00:00 app[web.1]: at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
> 2025-06-06T22:15:41.919359+00:00 app[web.1]: at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData)
> 2025-06-06T22:15:41.919359+00:00 app[web.1]: at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
> 2025-06-06T22:15:41.919360+00:00 app[web.1]: at Elsa.Secrets.Management.DataProtectionEncryptor.DecryptAsync(String encryptedValue, CancellationToken cancellationToken)
> 2025-06-06T22:15:41.919360+00:00 app[web.1]: at Elsa.Secrets.Management.DefaultSecretEncryptor.DecryptAsync(Secret secret, CancellationToken cancellationToken)
> 2025-06-06T22:15:41.919360+00:00 app[web.1]: at Elsa.Secrets.Api.Endpoints.Secrets.GetInputModel.Endpoint.ExecuteAsync(Request req, CancellationToken ct)
> 2025-06-06T22:15:41.919360+00:00 app[web.1]: at FastEndpoints.Endpoint`2.ExecAsync(CancellationToken ct)
> 2025-06-06T22:15:41.919360+00:00 app[web.1]: at FastEndpoints.Endpoint`2.ExecAsync(CancellationToken ct)
> 2025-06-06T22:15:41.919361+00:00 app[web.1]: at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
> 2025-06-06T22:15:41.919361+00:00 app[web.1]: at Elsa.Http.Middleware.HttpWorkflowsMiddleware.InvokeAsync(HttpContext httpContext, IServiceProvider serviceProvider)
> 2025-06-06T22:15:41.919362+00:00 app[web.1]: at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
> 2025-06-06T22:15:41.919362+00:00 app[web.1]: at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
> 2025-06-06T22:15:41.919362+00:00 app[web.1]: at NSwag.AspNetCore.Middlewares.SwaggerUiIndexMiddleware.Invoke(HttpContext context)
> 2025-06-06T22:15:41.919362+00:00 app[web.1]: at NSwag.AspNetCore.Middlewares.RedirectToIndexMiddleware.Invoke(HttpContext context)
> 2025-06-06T22:15:41.919363+00:00 app[web.1]: at NSwag.AspNetCore.Middlewares.OpenApiDocumentMiddleware.Invoke(HttpContext context)
> 2025-06-06T22:15:41.919363+00:00 app[web.1]: at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
> 2025-06-06T22:15:41.919472+00:00 app[web.1]: info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
> 2025-06-06T22:15:41.919473+00:00 app[web.1]: Executing endpoint '/_Host'

## Expected Behavior

- Possibly provide a convenience method or extension in Elsa.Secrets to configure key storage externally (like db, redis, etc) persistance somewhere.

CC: @sfmskywalker

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.