BUG: Eigenvectors generated via linalg.eigh are not orthonormal in v1.26.0

Describe the issue:

Hi,

While computing the eigenvectors of Hermitian matrices using numpy.linalg.eigh the eigenvectors are not coming out to be orthonormal. The eigenvectors are orthonormal using the scipy.linalg.eigh routine.

A jupyter notebook with a simple example is attached.

Compare_diag.pdf

Reproduce the code example:

import numpy as np
import scipy as sp

def create_random_hermitian_mat(dim):
    a = np.random.random((dim,dim)) + 1j*np.random.random((dim,dim))
    b = np.conj(a).T + a
    return b

a = create_random_hermitian_mat(4)
u, v = np.linalg.eigh(a)
print("Numpy Results = \n", np.matmul(np.conj(v).T, v))
u_sp, v_sp = sp.linalg.eigh(a)
print("\n Scipy results = \n", np.matmul(np.conj(v_sp).T, v_sp))

Output:

Numpy Results = 
[[ 2.   +0.j    0.   +0.j    -0.   -0.j    1.586+1.586j]
[ 0.   +0.j    2.   +0.j    0.   +0.j    -0.651-0.651j]
[-0.   +0.j    0.   -0.j    2.   +0.j    -1.027-1.027j]
[1.586-1.586j -0.651+0.651j -1.027+1.027j  5.   +0.j   ]]

Scipy Results =
[[ 1.+0.j -0.-0.j -0.-0.j  0.-0.j]
[-0.+0.j  1.+0.j  0.+0.j -0.+0.j]
[-0.+0.j  0.-0.j  1.+0.j  0.+0.j]
[ 0.+0.j -0.-0.j  0.-0.j  1.+0.j]]

Runtime information:

Numpy version.

'1.26.0'

Numpy configuration

Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/include
    lib directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/lib
    name: blas
    pc file directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/lib/pkgconfig
    version: 3.9.0
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/include
    lib directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/lib
    name: lapack
    pc file directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/lib/pkgconfig
    version: 3.9.0
Compilers:
  c:
    commands: arm64-apple-darwin20.0.0-clang
    linker: ld64
    name: clang
    version: 15.0.7
  c++:
    commands: arm64-apple-darwin20.0.0-clang++
    linker: ld64
    name: clang
    version: 15.0.7
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.0.2
Machine Information:
  build:
    cpu: aarch64
    endian: little
    family: aarch64
    system: darwin
  cross-compiled: true
  host:
    cpu: arm64
    endian: little
    family: aarch64
    system: darwin
Python Information:
  path: /Users/shinjan/Programs/miniforge3/envs/scicomp/bin/python
  version: '3.11'
SIMD Extensions:
  baseline:
  - NEON
  - NEON_FP16
  - NEON_VFPV4
  - ASIMD
  found:
  - ASIMDHP
  not found:
  - ASIMDFHM

Scipy Version

'1.11.3'

Scipy Configuration

Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/include
    lib directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/lib
    name: blas
    openblas configuration: unknown
    pc file directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/lib/pkgconfig
    version: 3.9.0
  lapack:
    detection method: pkgconfig
    found: true
    include directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/include
    lib directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/lib
    name: lapack
    openblas configuration: unknown
    pc file directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/lib/pkgconfig
    version: 3.9.0
  pybind11:
    detection method: pkgconfig
    include directory: /Users/shinjan/Programs/miniforge3/envs/scicomp/include
    name: pybind11
    version: 2.11.1
Compilers:
  c:
    commands: arm64-apple-darwin20.0.0-clang
    linker: ld64
    name: clang
    version: 15.0.7
  c++:
    commands: arm64-apple-darwin20.0.0-clang++
    linker: ld64
    name: clang
    version: 15.0.7
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 0.29.36
  fortran:
    commands: /Users/runner/miniforge3/conda-bld/scipy-split_1696467662374/_build_env/bin/arm64-apple-darwin20.0.0-gfortran
    linker: ld64
    name: gcc
    version: 12.3.0
  pythran:
    include directory: ../../_build_env/venv/lib/python3.11/site-packages/pythran
    version: 0.14.0
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: darwin
  cross-compiled: true
  host:
    cpu: arm64
    endian: little
    family: aarch64
    system: darwin
Python Information:
  path: /Users/shinjan/Programs/miniforge3/envs/scicomp/bin/python
  version: '3.11'

Context for the issue:

Erroneous eigenvectors from numpy.linalg.eigh

Start with the reproduced numpy.linalg.eigh example and compare its eigenvectors with scipy.linalg.eigh, as shown in the issue. Trace the eigh implementation and its numerical backend to identify why the returned vectors are not orthonormal; done means the example produces an orthonormal eigenvector matrix and regression coverage verifies it.