memory explosion with version > 0.3.10, (found through running scanpy).

Nobody has claimed this yet.

I'm running `scanpy (1.5.1)`.
I noticed that my memory usage was exploding when running their neighbor computing, filling up my 32GB ram.
My data has dimensions: `AnnData object with n_obs × n_vars = 38225 × 6829`

After some searching I narrowed it down to the call to `umap.nearest_neighbors`.

I had previously the version umap-learn `0.3.10` installed which does not have this issue. Any version I tested above that `0.4.0`, `0.4.2` and `0.4.5` all had the memory issue.

The umap-learn version `0.3.10` still has the `numba (0.50.1)` warnings but runs in the same if not faster than the later versions.

I'm not sure where or what to do to reproduce this bug but I ran the following code

```
sc.pp.pca(adata, n_comps=10)
sc.pp.neighbors(adata, n_neighbors=400, n_pcs=10)
```
I intentionally ran with few PCs becouse it seemed to trigger this issue but e.g. 200, 200 also triggers it.
This means of course that the data dimension input to umap is (38225x10) in this case.

Let me know if i can supply more info.
Thanks.

Start at the umap.nearest_neighbors entry point and reproduce the report using the scanpy calls sc.pp.pca and sc.pp.neighbors with the stated data dimensions and versions. Done means identifying and fixing the memory explosion seen in versions above umap-learn 0.3.10, with comparable behavior verified against the older version.

Written by the indexing model from the issue text.