michaelfeil / michaelfeil/infinity
Docs on disabling vector disk cache on high throughput scenario
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
## Request
I am suggesting/requesting a section documenting that disabling vector disk cache might bring significant performance boost, if the throughput is particularly high (if I am correct).
## Context
Please forgive me not following templates as this doesn't seem to fit into any.
I was trying to achieve a high-throughput embedding service on a powerful gpu instance (30k+ embeddings/sec), but the gpu usage never exceeds even 5% when I'm using a proper batch size (1536 in my case), and it hits 429 very quickly. For some reason the interval between each batch can take up to ~10 seconds.
After many attempts and troubleshooting I realized the vector disk cache might be the bottleneck here. In this case there was no benefits of keeping a cache of embeddings, and it would still be too intensive even if it's in a separate thread. After setting no-vector-disk-cache I was able to achieve 100% gpu usage.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the documentation for vector disk caching and the no-vector-disk-cache option. Confirm how the option affects high-throughput embedding workloads, then add a section describing when disabling the cache may help and what tradeoff it has; done means the guidance is clear and supported by the reported scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, performance
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100