jxnl / jxnl/instructor-classify
Support all Instructor clients in the unified evaluation framework
- Dominant language
- Python
- Stars
- 37
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
## Description
Currently, the unified evaluation framework (UnifiedEvaluator) only supports OpenAI as a client, but Instructor supports multiple LLM providers (Anthropic, Google, etc.). We should extend the evaluator to support all clients supported by Instructor.
## Current Limitation
In unified_eval.py, the evaluator directly initializes an OpenAI client:
```python
# Initialize OpenAI client with Instructor
client = instructor.from_openai(OpenAI())
```
## Proposed Solution
1. Add a configuration option for specifying the client type (OpenAI, Anthropic, Google, etc.)
2. Create a factory pattern for initializing the appropriate client based on the configuration
3. Update the InstrumentedClassifier to work with any client type
4. Ensure all client-specific code is properly abstracted
## Tasks
- [ ] Add client configuration to the YAML schema
- [ ] Create a client factory to instantiate the correct client type
- [ ] Update the async implementation to support different client types
- [ ] Add proper error handling for client initialization
- [ ] Add examples for different client types in the documentation
- [ ] Update tests to cover multiple client types
## Benefits
- Users can evaluate models across different providers in a single run
- More accurate cost comparisons between providers
- Consistent with Instructor's multi-provider philosophy
## Related
This is related to the parallelism enhancements, as both features improve the flexibility and utility of the evaluation framework.
Contributor guide
Assessment
This issue has not been assessed yet.