Question: Custom Service Provider Not Used Everywhere

So, I register my own IServiceProviderFactory using WebApplicationBuilder.Host.UseServiceProviderFactory, from which I return a custom IServiceProvider implementation. I see that it is called, and *appears* to work, however, when I try to inject some service into my controller, I see that it does not come from my IServiceProvider, but instead from an ServiceProviderEngineScope. Why is this? Shouldn't the registered factory be used for creating the global IServiceProvider?