Expand with cast causes an InvalidCastException

Assemblies affected
Microsoft.AspNetCore.OData 8.0.10
Microsoft.OData.Core 7.11.0

Describe the bug
Trying to cast in an expand, eg ~/Orders?$expand=Customer/Model.VipCustomer($orderby=VipEmail) (taken from https://github.com/OData/odata.net/pull/2300), results in an InvalidCastException:

An unhandled exception has occurred while executing the request.
     System.InvalidCastException: Unable to cast object of type 'Microsoft.OData.UriParser.TypeSegment' to type 'Microsoft.OData.UriParser.NavigationPropertySegment'.
        at Microsoft.AspNetCore.OData.Query.Validator.SelectExpandQueryValidator.ValidateRestrictions(Nullable`1 remainDepth, Int32 currentDepth, SelectExpandClause selectExpandClause, IEdmNavigationProperty navigationProperty, ODataValidationSettings validationSettings)
        at Microsoft.AspNetCore.OData.Query.Validator.SelectExpandQueryValidator.Validate(SelectExpandQueryOption selectExpandQueryOption, ODataValidationSettings validationSettings)
        at Microsoft.AspNetCore.OData.Query.SelectExpandQueryOption.Validate(ODataValidationSettings validationSettings)
        at Microsoft.AspNetCore.OData.Query.Validator.ODataQueryValidator.Validate(ODataQueryOptions options, ODataValidationSettings validationSettings)
        at Microsoft.AspNetCore.OData.Query.ODataQueryOptions.Validate(ODataValidationSettings validationSettings)
        at Microsoft.AspNetCore.OData.Query.EnableQueryAttribute.ValidateQuery(HttpRequest request, ODataQueryOptions queryOptions)
        at Microsoft.AspNetCore.OData.Query.EnableQueryAttribute.OnActionExecuting(ActionExecutingContext actionExecutingContext)
        at Microsoft.AspNetCore.Mvc.Filters.ActionFilterAttribute.OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
        at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
        at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
        at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
        at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
     --- End of stack trace from previous location ---
        at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
        at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
        at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
        at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
        at Microsoft.AspNetCore.OData.Routing.ODataRouteDebugMiddleware.Invoke(HttpContext context)
        at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

Trying to $filter for the correct type results in the same bug.

Reproduce steps
I have created a repo: https://github.com/andygjp/CastThenExpand

After starting the app, browse to https://localhost:7138/default/Emails?$expand=Customer/Default.VipCustomer and you'll get an InvalidCastException.

Browse to https://localhost:7138/default/Emails?$filter=isof(Customer,'Default.VipCustomer')&$expand=Customer/Default.VipCustomer and you'll get the same exception.

Data Model
See https://github.com/andygjp/CastThenExpand

EDM (CSDL) Model

<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
    <edmx:DataServices>
        <Schema Namespace="Default" xmlns="http://docs.oasis-open.org/odata/ns/edm">
            <EntityType Name="Customer">
                <Key>
                    <PropertyRef Name="ID"/>
                </Key>
                <Property Name="ID" Type="Edm.Int32" Nullable="false"/>
                <Property Name="Name" Type="Edm.String" Nullable="false"/>
                <Property Name="EmailID" Type="Edm.Int32"/>
                <NavigationProperty Name="Email" Type="Default.Email">
                    <ReferentialConstraint Property="EmailID" ReferencedProperty="ID"/>
                </NavigationProperty>
            </EntityType>
            <EntityType Name="VipCustomer" BaseType="Default.Customer">
                <Property Name="VipNumber" Type="Edm.Int32" Nullable="false"/>
            </EntityType>
            <EntityType Name="Email">
                <Key>
                    <PropertyRef Name="ID"/>
                </Key>
                <Property Name="ID" Type="Edm.Int32" Nullable="false"/>
                <Property Name="EmailAddress" Type="Edm.String" Nullable="false"/>
                <Property Name="CustomerID" Type="Edm.Int32"/>
                <NavigationProperty Name="Customer" Type="Default.Customer">
                    <ReferentialConstraint Property="CustomerID" ReferencedProperty="ID"/>
                </NavigationProperty>
            </EntityType>
            <EntityContainer Name="Container">
                <EntitySet Name="Customers" EntityType="Default.Customer">
                    <NavigationPropertyBinding Path="Email" Target="Emails"/>
                </EntitySet>
                <EntitySet Name="VipCustomers" EntityType="Default.VipCustomer">
                    <NavigationPropertyBinding Path="Email" Target="Emails"/>
                </EntitySet>
                <EntitySet Name="Emails" EntityType="Default.Email">
                    <NavigationPropertyBinding Path="Customer" Target="Customers"/>
                </EntitySet>
            </EntityContainer>
        </Schema>
    </edmx:DataServices>
</edmx:Edmx>

Expected behavior
It should filter the correct type and cast without the exception.