conversadocs / conversadocs/code-scan-docs

[Task]: llm abstraction

Open
#24 0 comments 0 reactions 1 assignee Claimed by @stoopidJSON View on GitHub
enhancement
Dominant language
Rust
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

### Description

Refactor the LLM integration in `csd_plugin_sdk` to provide a proper abstraction layer that supports multiple LLM providers beyond just Ollama. Create a provider-agnostic interface that allows plugins to use different LLM services (Ollama, OpenAI, Anthropic, etc.) through a common API, making it easy for users to switch providers or for developers to add new ones.

### Acceptance Criteria

- [x] **LLM Provider Abstraction**:
- Create abstract `LLMProvider` base class/trait defining the interface
- Move current Ollama implementation to `OllamaProvider` class
- Support provider-specific configuration while maintaining common interface
- Allow provider selection through configuration without code changes

- [x] **Provider-Agnostic Client**:
- `LLMClient` works with any provider implementing the interface
- Common methods for `generate()`, `test_connection()`, etc.
- Provider-specific features exposed through optional capabilities
- Consistent error handling across different provider types

- [ ] **Configuration Structure**:
- Support multiple provider configurations in `.csdrc.yaml`
- Provider type specified in config (e.g., `provider: "ollama"` or `provider: "openai"`)
- Provider-specific settings in nested configuration
- Easy switching between providers without plugin code changes

- [ ] **Token Management Abstraction**:
- Abstract token counting/estimation that providers can override
- Provider-specific model limits and capabilities
- Common interface for context window management
- Support for different tokenization strategies

- [ ] **Extensibility Framework**:
- Clear documentation on implementing new providers
- Example stub for OpenAI provider (not fully implemented)
- Provider registration/discovery mechanism
- Support for custom/third-party providers

- [ ] **Backward Compatibility**:
- Existing Ollama configurations continue to work
- Smooth migration path for current users
- Default to Ollama if provider not specified
- Clear deprecation warnings if using old config format

### Notes

This refactoring sets the foundation for supporting multiple LLM providers without requiring changes to output plugins. The abstraction should be simple enough for easy implementation of new providers while being flexible enough to handle provider-specific features. Focus on making the common case (generate text with context) simple while allowing advanced features through provider capabilities.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.