jxnl / jxnl/instructor-classify
Add Support for Local Models
- Dominant language
- Python
- Stars
- 37
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
## Description
Currently, the library primarily supports cloud-based LLM providers. Adding support for local model inference would expand the usability of the library, especially for users with privacy concerns or those working in environments without internet access.
## Tasks
- [ ] Implement support for local LLM inference
- [ ] Add adapters for frameworks like llama.cpp, Hugging Face Transformers, and similar local inference engines
- [ ] Support for quantized models to improve performance on consumer hardware
- [ ] Create examples demonstrating local model usage
- [ ] Document performance expectations and hardware requirements
- [ ] Add benchmarks comparing local models to cloud-based alternatives
- [ ] Ensure all evaluation tools work with local models
## Expected Result
Users should be able to run classifications using local models with the same API as cloud-based models. The library should provide guidance on model selection, quantization, and performance expectations.
## Why is this important?
Local model support increases the flexibility of the library, addresses privacy concerns, reduces costs for high-volume users, and enables usage in air-gapped environments.
## Difficulty
Hard - requires understanding of local inference engines and their integration with the existing architecture
Contributor guide
Assessment
This issue has not been assessed yet.