developmentseed / developmentseed/bioacoustics-api

Try using DiskANN

Open
#55 7 comments 0 reactions 0 assignees View on GitHub
maintenance
Dominant language
Jupyter Notebook
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Milvus supports on-disk index https://milvus.io/docs/disk_index.md. This should reduce compute usage drastically. DiskANN is enabled by default but I think we need to make sure the volume inside the container is mounted at `/var/lib/milvus/data`

I think we'll want to add a [non-boot persistent disk](https://cloud.google.com/compute/docs/disks/persistent-disks) to the cluster and then have k8s mount that to the query nodes and index nodes. We'll need to use `DISKANN` as the index type while building the index.

@leothomas Once you have the index update done #54, let's figure out a plan here. Might be good to create a second indexing script/job that can create the DiskANN index that has a volume mounted.

I should be able to quickly adjust the k8s configuration to support this and get some help @sunu too.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.