Add semantic LLM response caching (GPTCache-style) to reduce cost and latency

Is your feature request related to a problem?

Our current LLM usage re-sends repeated or highly similar prompts to the provider instead of reusing prior results. This increases token spend, adds avoidable latency, and puts more pressure on rate limits as usage grows. We do not currently have a shared semantic caching layer in front of LLM calls.

Describe the solution you'd like

Add a semantic cache layer in front of our LLM calls, similar to GPTCache, to reduce token usage, improve latency, and increase throughput.

For the first iteration, the implementation should:

• Introduce a single shared client or abstraction for LLM calls, such as LLMClient, so the rest of the codebase does not depend directly on caching details.
• Integrate a semantic cache behind that client that:
  • Uses embedding-based similarity search over prior queries
  • Reuses full LLM responses on cache hits
  • Supports configurable behavior such as similarity threshold, TTL, and backend
• Make caching easy to:
  • Enable or disable through config or environment variables
  • Swap out later, for example GPTCache versus a custom implementation
• Add minimal observability for:
  • Cache hits and misses
  • Latency impact
  • Reduction in upstream LLM calls where measurable

Acceptance criteria for an initial version:

• All LLM calls go through a shared client interface
• Repeated or semantically similar prompts can hit cache when enabled
• Cache behavior is configurable without code changes

Describe alternatives you've considered

• Continue without caching and accept the current cost and latency profile
• Add only exact-match request caching, which is simpler but would miss many near-duplicate prompts
• Build a custom semantic caching layer from scratch instead of using a GPTCache-style approach
• Limit caching to specific low-risk or high-volume call paths first

Additional context

This would be a strong infrastructure improvement for cost control and responsiveness, especially if we expect repeated research or assistant workflows to generate overlapping prompts.

Suggested first-iteration scope:

• Refactor LLM usage behind a single entrypoint
• Add semantic caching for basic chat/completion calls
• Add config flags for enable/disable and basic thresholds
• Add lightweight observability for cache hits, misses, and latency

Out of scope for now:

• Advanced eviction strategies
• Distributed caching
• Multi-tenant isolation
• Caching highly stateful or intentionally non-deterministic operations