Stopped running without error/warning

Hi there,

I've been trying to run umap on ~5.8 million data points on computer with 128GB RAM and 16 cores. I found my code to run successfully on a subset of 100 data points, but it will just stop running after couple of hours or so if i run it on the 5.8 million data points. Any clue as to why this is happening?

I cannot share my dataset unfortunately as it's confidential. However this is a snippet of my code:

data = pd.read_csv(data_fname)
sample = data['Sample']

data_features_only = data.iloc[:, :38]

data_train, data_test, sample_train, sample_test = train_test_split(data_features_only, sample)

reducer = umap.UMAP(n_epochs=200, low_memory=True).fit(data_train)
data_test_reduced = reducer.transform(data_test)

print(reducer.embedding_)
print(data_test_reduced)

The code didn't print out anything. Thus I presume it just fails at either the fit line or the transform line. Any advice as to how to debug this?

Thanks!

No repository file or test is identified. Start by isolating whether UMAP stops during fit or transform with the supplied snippet, then collect runtime and resource information for the 5.8-million-row case. Done means identifying a reproducible cause or a clearly scoped fix or workaround.