Falcon-11B support
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 537
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
Hi
I converted Falcon-11b using ct2-transformers-converter but I get the following error when trying to use the model.
Ctranslate version: 4.1.0
Conversion command:
ct2-transformers-converter --model tiiuae/falcon-11b --output_dir falcon-11b-base-ct2 --quantization int8 --trust_remote_code
Using model:
>>> import ctranslate2
>>> from transformers import AutoTokenizer
>>> model = ctranslate2.Generator("falcon-11b-base-ct2", device='cpu')
>>> tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-11b")
>>> outputs = model.generate_batch([tokenizer.convert_ids_to_tokens(tokenizer.encode("Falcon 11b is a new LLM"))], sampling_topk=10, max_length=200, include_prompt_in_result=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: axis 2 has dimension 6144 but expected 4352
>>>
I assume this is due to the new model being unsupported, will falcon 11b be supported by CT2?
Thanks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the ct2-transformers-converter command and the ctranslate2.Generator example from the issue with Falcon-11B. Trace the conversion and generation entry points to identify where the 6144-versus-4352 dimension mismatch occurs. Done means Falcon-11B converts and generates successfully with the reported configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100