microsoft / microsoft/aspire

Aspire do support for run mode related artifacts

Open
#18,971 3 comments 0 reactions 0 assignees View on GitHub
area-app-model area-pipelines triage:bot-seen
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Is your feature request related to a problem? Please describe the problem.

We run the apphost in CI using `aspire start` and then `aspire wait` on the critical resources.
But we kept on having timing issues due to network pulls of the different images from registry.

So we created a pipeline step to pre-pull all the runtime images.

However, the resource model in pipeline steps is totally different than in run mode, causing it to be harder to author, but not only that, when we will start to publish using `aspire publish` all the local emulators and prepackaged services (like redis) will be replaced with terraform and helm charts and not containers.
So we will lose the ability to use the apphost to tell us what the actual list of images needed to be pulled is.

### Describe the solution you'd like

I want to be able to register pipeline steps that execute with the `aspire run` context - access to the actual AppModel we use during run mode, and not publish mode.

### Additional context

Here is the snippet we had to implement and see how ackward it is to figure out what is an image and what is an executable and should not be pulled:

```csharp
// Prevents network saturation and Docker Hub / registry rate-limit errors.
var pullSemaphore = new SemaphoreSlim(maxConcurrentPulls, maxConcurrentPulls);

builder.Pipeline.AddStep(
name: "prepull-images",
action: ctx =>
{
ctx.Logger.LogInformation("prepull-images: all image pulls completed.");
return Task.CompletedTask;
});

builder.Pipeline.AddPipelineConfiguration(async configCtx =>
{
var existingStepNames = new HashSet(
configCtx.Steps.Select(s => s.Name),
StringComparer.OrdinalIgnoreCase);

var seenImages = new HashSet(StringComparer.OrdinalIgnoreCase);

foreach (var resource in configCtx.Model.Resources)
{
if (!resource.TryGetLastAnnotation(out var img))
{
continue;
}

if (resource.HasAnnotationOfType())
{
continue;
}

var hasRegistry = !string.IsNullOrWhiteSpace(img.Registry);
var registry = hasRegistry ? img.Registry!.TrimEnd('/') + "/" : "";
var tag = string.IsNullOrWhiteSpace(img.Tag) ? "latest" : img.Tag;
var imageRef = $"{registry}{img.Image}:{tag}";
var stepName = $"pull-{resource.Name}";

// Skip if already registered (guard against multiple invocations).
if (existingStepNames.Contains(stepName) || !seenImages.Add(imageRef))
{
continue;
}

string? platform = null;
foreach (var annotation in resource.Annotations.OfType())
{
var callbackCtx = new ContainerRuntimeArgsCallbackContext([], CancellationToken.None);
await annotation.Callback(callbackCtx);
var args = callbackCtx.Args.Select(a => a.ToString() ?? "").ToList();
var platformIdx = args.IndexOf("--platform");
if (platformIdx >= 0 && platformIdx + 1 < args.Count)
{
platform = args[platformIdx + 1];
break;
}
}

var platformCapture = platform;

builder.Pipeline.AddStep(
name: stepName,
action: async ctx =>
{
var platformArg = platformCapture is not null ? $"--platform {platformCapture} " : "";

// Throttle concurrent pulls to avoid network saturation and registry rate-limits.
await pullSemaphore.WaitAsync(ctx.CancellationToken);
try
{
ctx.Logger.LogInformation("Pulling {Image}...", imageRef);
var sw = Stopwatch.StartNew();

var psi = new ProcessStartInfo("docker", $"pull --quiet {platformArg}{imageRef}")
{
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false,
};

using var proc = Process.Start(psi)
?? throw new InvalidOperationException($"Failed to start 'docker pull' for {imageRef}");

await proc.WaitForExitAsync(ctx.CancellationToken);
sw.Stop();

if (proc.ExitCode != 0)
{
var stderr = await proc.StandardError.ReadToEndAsync(ctx.CancellationToken);
throw new InvalidOperationException(
$"docker pull failed for {imageRef} (exit {proc.ExitCode}): {stderr.Trim()}");
}

ctx.Logger.LogInformation("✓ {Image} pulled in {Elapsed:F1}s", imageRef, sw.Elapsed.TotalSeconds);
}
finally
{
pullSemaphore.Release();
}
},
requiredBy: new[] { "prepull-images" });
}
});
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing how `aspire run` builds the AppModel and how `AddPipelineConfiguration` currently receives the publish-mode resource model. Compare the run and publish contexts, including `ContainerImageAnnotation` and runtime arguments. Done means pipeline steps can access the run-mode AppModel and identify the runtime images needed for pre-pulling without relying on publish-mode resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, docker
Domain
ci-cd, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.