inv(Inf*I) returns Nan for dims between (2,2) and (4,4)

When calculating the inverse of a matrix with inf-entries, StaticArrays returns NaNs:

julia> inv(SMatrix{3,3}(Inf*I))
3×3 SMatrix{3, 3, Float64, 9} with indices SOneTo(3)×SOneTo(3):
 NaN  NaN  NaN
 NaN  NaN  NaN
 NaN  NaN  NaN

Using normal matrices results in zero:

julia> inv(Inf*I(3))       # Diagonal
3×3 Diagonal{Float64, Vector{Float64}}:
 0.0   ⋅    ⋅ 
  ⋅   0.0   ⋅ 
  ⋅    ⋅   0.0
julia> inv(Inf*Matrix(I(3)))  # Also for full matrix
3×3 Matrix{Float64}:
 0.0  0.0  0.0
 0.0  0.0  0.0
 0.0  0.0  0.0

This is true for the Dims (2,2) (3,3) and (4,4) and probably due to the specialized methods for these sizes in inv.jl.

Reproduce the issue in Julia for the 2×2, 3×3, and 4×4 cases, then inspect the specialized inverse methods in inv.jl. Compare their behavior with inv(Inf*I(N)) for normal matrices; done means the StaticArrays results no longer produce NaNs for these diagonal infinity inputs.