WebApplicationFactory Client returns NotFound for all requests with Overriding Configure method

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

Hi,
I create bellow [custom factory](https://github.com/mehdihadeli/e2e-test/blob/master/Test/CustomWebApplicationFactory.cs) for E2E test:

``` csharp
public class CustomWebApplicationFactory : WebApplicationFactory
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureServices(services =>
{
// extend service collection
});

builder.ConfigureTestServices(services =>
{
// extend service collection
});

builder.Configure(app =>
{
// extend application builder
});
}
}
```
Also, this is my [Program](https://github.com/mehdihadeli/e2e-test/blob/master/App/Program.cs) file:

``` csharp
using System.Reflection;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.

builder.Services.AddControllers() .AddApplicationPart(Assembly.GetExecutingAssembly());;
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
app.UseSwagger();
app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.MapGet("WeatherForecast", _ =>
{
var summaries = new[]
{
"Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
};
return Task.FromResult(Enumerable.Range(1, 5).Select(index => new WeatherForecast
{
Date = DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
TemperatureC = Random.Shared.Next(-20, 55),
Summary = summaries[Random.Shared.Next(summaries.Length)]
}));
});

app.Run();

public partial class Program{}
```

When I want to call `WeatherForecast` GET endpoint, with WebApplicationFactory `Client` I get `NotFound` response from client in my test instead of `OK`

``` csharp
public class UnitTest1: CustomWebApplicationFactory
{
[Fact]
public async Task Test1()
{
var response = await CreateClient().GetAsync("/WeatherForecast");
response.Should().Be200Ok();
}
}
```
If I remove below section from My CustomFactory my http client returns OK status code

``` csharp
builder.Configure(app =>
{
// extend application builder
});
```

### Expected Behavior

I Expect to get a 200 ok status code for this http call on http client

### Steps To Reproduce
Run [this test](https://github.com/mehdihadeli/e2e-test/blob/master/Test/UnitTest1.cs#L8)

### Exceptions (if any)

``` bash
Xunit.Sdk.XunitException
Expected response to be HttpStatusCode.OK {value: 200}, but found HttpStatusCode.NotFound {value: 404}.

The HTTP response was:

HTTP/1.1 404 NotFound
Content-Length: 0

The originated HTTP request was:

GET http://localhost/WeatherForecast HTTP 1.1
Content-Length: 0

at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args)
at FluentAssertions.Web.HttpResponseMessageAssertions.Be200Ok(String because, Object[] becauseArgs)
at FluentAssertions.HttpStatusCodeAssertionsExtensions.Be200Ok(HttpResponseMessageAssertions`1 parent, String because, Object[] becauseArgs)
at E2E.UnitTest1.Test1() in C:\Users\MehdiHadeli\Desktop\E2E\Test\UnitTest1.cs:line 11
at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_0.<b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 264
--- End of stack trace from previous location ---
at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90
```

### .NET Version

7.0.0

### Anything else?

_No response_