arc53 / arc53/DocsGPT

fix: AnthropicLLM uses deprecated completions API, broken for Claude 3+ and drops full conversation history

Open
#2,564 1 comment 1 reaction 1 assignee Claimed by @immuhammadfurqan View on GitHub
Dominant language
Python
Stars
18.3k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
29

Description

## Bug Description

`application/llm/anthropic.py` uses the legacy Claude 2-era text completions API:

```python
# lines 47-52
completion = self.anthropic.completions.create(
model=model,
max_tokens_to_sample=max_tokens,
prompt=f"{self.HUMAN_PROMPT} {prompt}{self.AI_PROMPT}",
)
```

This API is not supported by any Claude 3/3.5/4 model. Additionally both `_raw_gen` and `_raw_gen_stream` only extract `messages[0]` and `messages[-1]`, silently discarding the entire multi-turn conversation history between them. All other LLM providers pass the full `messages` list.

## Expected Behavior

`AnthropicLLM` should use `client.messages.create()` / `client.messages.stream()` (the modern Messages API), pass the full conversation history, and correctly handle system messages via the `system=` parameter.

## Proposed Fix

- Remove deprecated `HUMAN_PROMPT`/`AI_PROMPT` imports and instance variables
- Rewrite `_raw_gen` to use `anthropic.messages.create()`, return `response.content[0].text`
- Rewrite `_raw_gen_stream` to use `anthropic.messages.stream()`, yield text deltas
- Add `_split_messages()` helper to separate system prompt from conversation turns

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.