lmcinnes / lmcinnes/pynndescent
Speed-up Indexing with custom metrics
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 970
- Forks
- 107
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thanks for the great package! Currently I'm trying to build a kNN with custom metrics for DTW, but the index building for just 90k of data would take around 5 hours. Do you have any suggestions to improve this?
Below is the code that I used to build the index and perform a query
@jit(nopython=True, fastmath=True)
def dtw_numba(x, y):
"""
Compute the Dynamic Time Warping (DTW) distance between two sequences.
Parameters:
x : array-like
First sequence.
y : array-like
Second sequence.
Returns:
float
The DTW distance between sequences x and y.
"""
n, m = len(x), len(y)
dtw_matrix = np.full((n + 1, m + 1), np.inf)
dtw_matrix[0, 0] = 0
for i in range(1, n + 1):
for j in range(1, m + 1):
cost = (x[i - 1] - y[j - 1]) ** 2
dtw_matrix[i, j] = cost + min(dtw_matrix[i - 1, j], # insertion
dtw_matrix[i, j - 1], # deletion
dtw_matrix[i - 1, j - 1]) # match
return np.sqrt(dtw_matrix[n, m])
import pynndescent
index = pynndescent.NNDescent(flat_inj_vecs, metric=dtw_numba)
index.query([flat_vecs[10]], k=100)
Thank you!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by profiling the dtw_numba metric during NNDescent construction and the subsequent index.query call, using the 90k-data example as a benchmark. Determine whether the indexing cost comes from the custom DTW metric or NNDescent's metric integration; done means a documented, reproducible improvement or a clearly supported limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100