lmcinnes / lmcinnes/pynndescent

Is it possible to speed up index generation for time series data?

Open
#174 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
970
Forks
107
PR merge metrics
No merged PRs in 30d

Description

I have time series data that I need to query, but queries have to be done separately for each time point.
Here's my current approach using toy data:
```python
# m is the number of measurements recorded
# d are all recorded data points for each m
# s are separate data points (e.g. from another experiment) we need to query over d for each m
m, d, s = 10, 10000, 10
t = np.random.random(m * d * 3).reshape(m, d, 3).astype('float32')
q = np.random.random(m * s * 3).reshape(m, s, 3).astype('float32')
for mi in range(m):
xb, xq = t[mi, :], q[mi, :]
index = pynndescent.NNDescent(xb) # n_jobs does not seem to make a difference.
index.prepare()
ii, dd = index.query(xq)
# ... process ii and dd
# Takes around 23 seconds
```
Setting up and preparing the index takes up the vast majority of the processing time.
Is it possible to speed up this type of queries? The key is that queries have to be done for each `t(i)` and `q(i)` pair separately. In our experiments, we can have `m` in the range `10**4` or more.
`NNDescent` takes `n_jobs` as arguments, but I do not observe a performance difference with the above code.
If I, instead, use `joblib` for this, I do get a noticeable performance improvement:
```python
def profile_pynnd_joblib(xb, xq):
index = pynndescent.NNDescent(xb)
index.prepare()
ii, dd = index.query(xq)
return None
# joblib parallelization:
from joblib import Parallel, delayed
Parallel(n_jobs=10)(delayed(prof_pynnd_joblib)(t[mi, :], q[mi, :]) for mi in range(m))
# Takes around 2.5 seconds
```
Because each measurement `m` is independent, this parallelization makes sense and is obvious. But I am wondering if there is a smarter way to gain performance improvements. Perhaps in preparing the index, but also perhaps being smarter in how the index is generated? As I mentioned, `m` can be quite large for us, so, unfortunately, neither of these approaches are currently viable.
Suggestions are very welcome.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducible NNDescent and joblib examples in the issue, profiling index construction, prepare(), and query() separately. Compare the serial and parallel timings across larger m values; done means identifying and documenting a viable performance improvement or confirming that the current approach requires project-level design work.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.