jxnl / jxnl/instructor-classify

Support all Instructor clients in the unified evaluation framework

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
37
Forks
4
PR merge metrics
No merged PRs in 30d

Description

## Description
Currently, the unified evaluation framework (UnifiedEvaluator) only supports OpenAI as a client, but Instructor supports multiple LLM providers (Anthropic, Google, etc.). We should extend the evaluator to support all clients supported by Instructor.

## Current Limitation
In unified_eval.py, the evaluator directly initializes an OpenAI client:
```python
# Initialize OpenAI client with Instructor
client = instructor.from_openai(OpenAI())
```

## Proposed Solution
1. Add a configuration option for specifying the client type (OpenAI, Anthropic, Google, etc.)
2. Create a factory pattern for initializing the appropriate client based on the configuration
3. Update the InstrumentedClassifier to work with any client type
4. Ensure all client-specific code is properly abstracted

## Tasks
- [ ] Add client configuration to the YAML schema
- [ ] Create a client factory to instantiate the correct client type
- [ ] Update the async implementation to support different client types
- [ ] Add proper error handling for client initialization
- [ ] Add examples for different client types in the documentation
- [ ] Update tests to cover multiple client types

## Benefits
- Users can evaluate models across different providers in a single run
- More accurate cost comparisons between providers
- Consistent with Instructor's multi-provider philosophy

## Related
This is related to the parallelism enhancements, as both features improve the flexibility and utility of the evaluation framework.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.