huggingface / huggingface/candle
How to run LLama-3 or Phi with more then 4096 prompt tokens?
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Could you please show me an example where LLama-3 model used (better GGUF quantized) and initial prompt is more then 4096 tokens long? Or better 16-64K long (for RAG). Currently everything I do ends with error:
In this code:
let logits = model.forward(&input, 0); // input is > 4096 tokens
Error:
narrow invalid args start + len > dim_len: [4096, 64], dim: 0, start: 0, len:4240
Model used:
https://huggingface.co/MaziyarPanahi/Llama-3-8B-Instruct-64k-GGUF
Thank you a lot in advance!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the Rust model.forward call shown in the issue and trace where the 4096-token dimension is imposed for the linked GGUF model. Verify whether the result should be a working long-context example or documentation of the supported limit, then ensure the chosen outcome addresses prompts beyond 4096 tokens.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100