WorkflowHost Completes Exit Before Steps Within Workload Can Attempt To Stop Gracefully

Written on issue #645, but created a new issue because it may be unrelated.

I expect WorkflowHost not to complete its Stop lifecycle hooks until after its Steps have been given a chance to complete. WorkflowHost instead seems to exit as soon as the signal is sent, preventing steps from finishing their work / their cleanup.

The ability to do that cleanup is more important to me than the specifics of implementation, so if there is a better way to have the behavior that we can carry out some process once WorkflowHost receives a graceful stop signal, I'd appreciate that, too.

> This appears to still be occurring on 3.17.0. I am completely prepared for a response that we are using WorkflowCore essentially incorrectly, but I think the behavior I'm seeing is unexpected. I want my step processes to complete or at least attempt to complete when the WorkflowHost is stopped. Instead, that stop seems to try exiting the steps completely without any lifecycle hooks running.
>
> I install and run the following service in Windows. No matter when along the path I stop the service, I would expect MyInnerService to attempt its StopAsync. It does not. I only see the Exec and Start logs. I also tried replacing the hosted MyInnerService with a simple Task.Delay in the step. In that case, too, the Step does not let the delay complete and the workload simply exits out when I stop the service.
>
> Is that expected behavior?
>
> ```
> Directory.SetCurrentDirectory(SharedPaths.BaseDataDirectory);
>
> var bootstrapBuilder = Host.CreateApplicationBuilder();
>
> var configuration = new ConfigurationBuilder()
> .SetBasePath(AppDomain.CurrentDomain.BaseDirectory)
> .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true).Build();
>
> bootstrapBuilder.Services.AddSingleton(configuration);
>
> bootstrapBuilder.Services.AddWorkflow();
>
> bootstrapBuilder.Services
> .AddTransient()
> .AddHostedService();
>
> bootstrapBuilder.Services
> .AddLogging(
> options =>
> {
> var logFile = string.Format("Bootstrap.log");
> var logPath = Path.Combine(SharedPaths.CommonLogDirectory, logFile);
> var logger = new LoggerConfiguration()
> .ReadFrom.Configuration(bootstrapBuilder.Configuration)
> .WriteTo.File(logPath, formatProvider: CultureInfo.InvariantCulture, rollingInterval: RollingInterval.Day, retainedFileCountLimit: 15, shared: true)
> .WriteTo.Console(formatProvider: CultureInfo.InvariantCulture)
> .CreateLogger();
> options
> .ClearProviders()
> .AddSerilog(logger);
> });
>
>
> bootstrapBuilder.Services
> .AddWindowsService();
>
> var bootstrapper = bootstrapBuilder.Build();
>
> var host = bootstrapper.Services.GetRequiredService();
> host.Options.UseMaxConcurrentWorkflows(1);
> host.RegisterWorkflow();
>
> await host.StartWorkflow("Workflow", 1, new MyBootstrapState { });
>
> await bootstrapper.RunAsync();
>
> public class MyWorkflow : IWorkflow
> {
> ///
> public string Id => "Workflow";
>
> ///
> public int Version => 1;
>
> ///
> public void Build(IWorkflowBuilder builder)
> {
> builder
> .While(state => true)
> .Do(recur => recur.Saga(mainSaga =>
> {
> mainSaga
> .StartWith();
> }));
> }
> }
>
> public class MyStep : StepBodyAsync
> {
> public override async Task RunAsync(IStepExecutionContext context)
> {
> var bootstrapState = (MyBootstrapState)context.Workflow.Data;
>
> var webBuilder = WebApplication.CreateBuilder();
> webBuilder.Services
> .AddLogging(configure =>
> {
> var logPath = Path.Combine(SharedPaths.CommonLogDirectory, GetLogFileName());
> var logger = new LoggerConfiguration()
> .ReadFrom.Configuration(webBuilder.Configuration)
> .WriteTo.File(logPath, formatProvider: CultureInfo.InvariantCulture, rollingInterval: RollingInterval.Day, retainedFileCountLimit: 15, shared: true)
> .WriteTo.Console(formatProvider: CultureInfo.InvariantCulture)
> .CreateLogger();
> configure
> .ClearProviders()
> .AddSerilog(logger);
> });
>
> webBuilder.Services.AddHostedService();
>
> await webBuilder.Build().RunAsync();
>
> return ExecutionResult.Next();
> }
>
> public static string GetLogFileName()
> {
> return string.Format("My_Step.log");
> }
> }
>
> public class MyInnerService : BackgroundService
> {
> public ILogger _logger;
> public MyInnerService(ILogger logger) { _logger = logger; }
>
> protected override async Task ExecuteAsync(CancellationToken stoppingToken)
> {
> _logger.LogInformation("STARTING EXEC INNER SERVICE");
> await Task.Delay(5000, stoppingToken);
> _logger.LogInformation("ENDING EXEC INNER SERVICE");
> }
>
> public override async Task StartAsync(CancellationToken stoppingToken)
> {
> _logger.LogInformation("STARTING START INNER SERVICE");
> await Task.Delay(5000, stoppingToken);
> _logger.LogInformation("ENDING START INNER SERVICE");
> }
>
> public override async Task StopAsync(CancellationToken stoppingToken)
> {
> _logger.LogInformation("STARTING STOP INNER SERVICE");
> await Task.Delay(5000, stoppingToken);
> _logger.LogInformation("ENDING STOP INNER SERVICE");
> }
> }
>
> public class MyBootstrapState
> {
> }
> ```

_Originally posted by @DelineaSamGeorge in [#645](https://github.com/danielgerlag/workflow-core/issues/645#issuecomment-3398440757)_

Reproduce the Windows service shutdown using the shown WorkflowHost, MyStep, and MyInnerService example, starting with the stop path between AddHostedService(), StepBodyAsync.RunAsync, and BackgroundService.StopAsync. Trace whether the workflow and step receive a graceful cancellation opportunity; done means stopping the host allows the step's cleanup or StopAsync work to be attempted before WorkflowHost completes its stop lifecycle.

Written by the indexing model from the issue text.