'UseExceptionHandler' doesn't work with OData?

I was trying to setup my API to redirect exceptions to an exception handler controller by using app.UseExceptionHandler("/error"), however whenever an error happens, I get a 404 instead.

Why am I getting a 404? I can manually navigate to "myApiUrl/error" and hit the error controller just fine, but when the error happens inside an OData controller in the application, it does not reach the error controller and produces 404 instead.

Here is a sample of my startup:

        public static void Configure(
            IApplicationBuilder app,
            IEdmModelProvider edmModelProvider)
        {
            app
                .UseExceptionHandler("/error")
                .UseHttpsRedirection()
                .UseRouting()
                .UseAuthorization()
                .UseCors(builder => builder.AllowAnyHeader().AllowAnyOrigin().AllowAnyMethod())
                .UseEndpoints(endpoints => endpoints.MapOData(edmModelProvider))
                .UseSwagger()
                .UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "MyApi"));
        }

And here is the error controller:

    [ApiController]
    [ApiExplorerSettings(IgnoreApi = true)]
    public class ErrorController : ControllerBase
    {
        [Route("/error")]
        public IActionResult Error() => Problem();
    }

When I transfer this into a standard MVC application, it works just fine, so I suspect it must be something to do with OData that I'm missing or doesn't work properly.