ContainerExecutableResource

@karolz-ms @danegsta and I had a design conversation this morning about `AddContainerExecutable(...)` and `ContainerExecutableResource` and how it behaves in non-persistent and persistent scenarios. What follows is a capture of some API surface designs from that conversation.

# Non-persistent scenarios

## Basic usage

The first thing that we discussed is what is the expected behavior of `AddContainerExecutable(...)` when it is applied to a container resource in the following manner.

```csharp
var builder = DistributedApplication.CreateBuilder(args);
builder.AddOllama("ollama")
.AddContainerExecutable("pull-model", "ollama", "/", ["pull", "llama2"]);
builder.Build().Run();
```

In this scenario DCP would start the Ollama container and as soon as it was running immediately start the container executable. It would not wait for the Ollama resource to become healthy because the assumption is that the container executable may influence the health of the parent resource.

# Basic usage with `WaitFor(...)` and friends

`ContainerExecutableResource` would also participate in the `WaitFor(...)` mechanics in Aspire. So it would be possible to do this:

```csharp
var builder = DistributedApplication.CreateBuilder(args);
var ollama = builder.AddOllama("ollama");
ollama.AddContainerExecutable("pull-model", "ollama", "/", ["pull", "llama2"])
.WaitFor(ollama);
builder.Build().Run();
```

In this scenario the `ContainerResourceExecutable` would not be started (the DCP resource would not be created) until the parent resource became healthy.

# Basic usage with explicit start

`ContainerResourceExecutable` would honor `WithExplicitStart(...)`. This is useful for scenarios where the container executable represents a command that you might want to do occasionally (like truncate a database) or hide behind a dashboard command and trigger via `ResourceCommandService`.

```csharp
var builder = DistributedApplication.CreateBuilder(args);
builder.AddOllama("ollama")
.AddContainerExecutable("clear-models", "ollama", "/", ["rm", "-a"])
.WithExplicitStart();
builder.Build().Run();
```

# Persistent scenarios

## Container executable with persistent parent container

When `ContainerExecutableResource` is used with `ContainerResource` that has a persistent lifetime annotation, the default behavior is that the executable will run again each time the apphost starts, regardless of whether the container is reused or not. This will work well for commands which are idempotent.

```csharp
var builder = DistributedApplication.CreateBuilder(args);
builder.AddOllama("ollama")
.WithLifetime(ContainerLifetime.Persistent)
.AddContainerExecutable("pull-model", "ollama", "/", ["pull", "llama2"]);
builder.Build().Run();
```

## Controlling execution based on parent container reuse

There would be a new extension method called `WithPersistenceBehavior(...)` which is applied to `ContainerExecutableResource` builders which sets an annotation on the resource which controls how it behaves when containers are reused. Here is an example.

```csharp
var builder = DistributedApplication.CreateBuilder(args);
builder.AddOllama("ollama")
.WithLifetime(ContainerLifetime.Persistent)
.AddContainerExecutable("pull-model", "ollama", "/", ["pull", "llama2"])
.WithPersistenceBehavior(PersistenceBehavior.SkipOnReuse);
builder.Build().Run();
```

There would be two enumerated behaviors, `RunWhenReused`, `RunWhenNew`, and `RunAlways' (default) - the enumerated names, extension methods etc are up for debate but the idea is that you might have commands that you want to execute depending on whether a container is fresh or not each time you start the app host.

Under the covers DCP would do house keeping to figure out whether specific command had been run against a container to determine whether it needs to run a specific command again. This would be similar to the fingerprinting mechanism that we have in DCP today for container reuse although @karolz-ms and @danegsta are considering what storage options we might have from this ranging from global container volumes which we `docker cp` state into to sqllite databases for tracking information. This would mostly be an implementation detail but some information my back-propogate into the app model.

> NOTE: When discussing with @karolz-ms and @danegsta we discussed reusing the method `WithLifetime(...)`. I've used `WithPersistenceBehavior(...)` here - but we could make it with `WithLifetime(ContainerExecutableLifetime)`.

## Persistent parent with WaitFor(...)

`WaitFor(...)` would work as expected for persistent parents as well. Basically it would wait until the app model recognized the parent resource as being healthy and then trigger the execution of the container executable via DCP.

```csharp
var builder = DistributedApplication.CreateBuilder(args);
var ollama = builder.AddOllama("ollama")
.WithLifetime(ContainerLifetime.Persistent);
ollama.AddContainerExecutable("pull-model", "ollama", "/", ["pull", "llama2"])
.WaitFor(ollama);
builder.Build().Run();
```

## Persistent parent with explicit start

If the parent resource is persistent, and the container executable is marked with explicit start, the container executable would not start unless explicitly started.

```csharp
var builder = DistributedApplication.CreateBuilder(args);
builder.AddOllama("ollama")
.WithLifetime(ContainerLifetime.Persistent)
.AddContainerExecutable("clear-models", "ollama", "/", ["rm", "-a"])
.WithExplicitStart();

builder.Build().Run();
```

If combined with `WithPersistenceBehavior(...)` then this would be considered an error and throw!!!

```csharp
var builder = DistributedApplication.CreateBuilder(args);
builder.AddOllama("ollama")
.WithLifetime(ContainerLifetime.Persistent)
.AddContainerExecutable("clear-models", "ollama", "/", ["rm", "-a"])
.WithExplicitStart()
.WithPersistenceBehavior(PersistenceBehavior.SkipOnReuse); // Should throw!

builder.Build().Run();
```

# Deployment considerations

## `ContainerExecutableResources` ... local first ... do not equal init containers

`ContainerExecutableResource` is a local only concept which unlike their container parents does not automatically translate well to common target deployment environments.

For example `Azure Container Apps` does not support defining a container app which also specifies executables that launch after the main container has started. ACA and Kubernetes do however support init containers that allow the developer to specify containers that need to run and complete successfully before the specified container in the pod starts.

`ContainerExecutableResource` IS NOT the same as an init container. An init container brings its own environment and optionally shares volumes with the container that will launch after it completes successfully. By contrast `ContainerExecutableResource` represents a command that runs INSIDE a container. Its an analogue to `docker exec` or `kubectl exec`.

For local development an extension developer may choose to provide extensibility functionality (such as installing plugins/installing models/creating databases) via `ContainerExexecutableResource` but they would need to find an alternative approach for deployment scenarios.

For example, if we had an API like `AddGrafana(string)` we might have extension methods like `WithPlugin(string)`. An example implementation for this would add annotations to the grafana resource to keep track of plugins that are registered, and then the apphost start it would dynamically add to `ContainerExecutableResource` resources to run the install commands.

For deployment, rather than doing plugin installs that way you might generate a Dockerfile for the container resource and add the install commands to the docker file. Avoiding the cost of doing a dockerfile build for local dev, but baking a container image for deployment purposes.