huggingface / huggingface/candle

How to run LLama-3 or Phi with more then 4096 prompt tokens?

Open
#2,171 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Could you please show me an example where LLama-3 model used (better GGUF quantized) and initial prompt is more then 4096 tokens long? Or better 16-64K long (for RAG). Currently everything I do ends with error:
In this code:
let logits = model.forward(&input, 0); // input is > 4096 tokens

Error:
narrow invalid args start + len > dim_len: [4096, 64], dim: 0, start: 0, len:4240

Model used:
https://huggingface.co/MaziyarPanahi/Llama-3-8B-Instruct-64k-GGUF

Thank you a lot in advance!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Rust model.forward call shown in the issue and trace where the 4096-token dimension is imposed for the linked GGUF model. Verify whether the result should be a working long-context example or documentation of the supported limit, then ensure the chosen outcome addresses prompts beyond 4096 tokens.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.