jxnl / jxnl/instructor-classify

Add Support for Local Models

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
37
Forks
4
PR merge metrics
No merged PRs in 30d

Description

## Description
Currently, the library primarily supports cloud-based LLM providers. Adding support for local model inference would expand the usability of the library, especially for users with privacy concerns or those working in environments without internet access.

## Tasks
- [ ] Implement support for local LLM inference
- [ ] Add adapters for frameworks like llama.cpp, Hugging Face Transformers, and similar local inference engines
- [ ] Support for quantized models to improve performance on consumer hardware
- [ ] Create examples demonstrating local model usage
- [ ] Document performance expectations and hardware requirements
- [ ] Add benchmarks comparing local models to cloud-based alternatives
- [ ] Ensure all evaluation tools work with local models

## Expected Result
Users should be able to run classifications using local models with the same API as cloud-based models. The library should provide guidance on model selection, quantization, and performance expectations.

## Why is this important?
Local model support increases the flexibility of the library, addresses privacy concerns, reduces costs for high-volume users, and enables usage in air-gapped environments.

## Difficulty
Hard - requires understanding of local inference engines and their integration with the existing architecture

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.