Different connectivties and UMAP if using data in X or obsm

• [ x] I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of scanpy.
• (optional) I have confirmed this bug exists on the master branch of scanpy.
  Version: '1.8.0.dev70+g560bd5d3'

I have made adata with same data in X and obsm. Then I used either of those two matrices for neighbours and subsequent UMAP. The connectives and UMAPs however look different despite same starting data (does not seem to be due to randomness as I get same UMAPs with same representations on multiple runs).

# Adata
adata_pb=sc.AnnData(latent_pb)
adata_pb.obsm['X_integrated']=latent_pb

# From rep=X
sc.pp.neighbors(adata_pb,n_pcs=0,use_rep='X')
c_x=adata_pb.obsp['connectivities']
sc.tl.umap(adata_pb)
rcParams['figure.figsize']=(6,6)
sc.pl.umap(adata_pb)

# From rep=obsm
sc.pp.neighbors(adata_pb,n_pcs=0,use_rep='X_integrated')
c_o=adata_pb.obsp['connectivities']
sc.tl.umap(adata_pb)
rcParams['figure.figsize']=(6,6)
sc.pl.umap(adata_pb)

np.allclose(c_x.A, c_o.A)
False

Start with the sc.pp.neighbors calls using use_rep='X' and use_rep='X_integrated', then follow how sc.tl.umap consumes the resulting connectivities. Reproduce the example with identical data in X and obsm, compare c_x.A with c_o.A and the resulting UMAPs, and consider the issue done when equivalent representations produce equivalent results.