OpenNMT / OpenNMT/CTranslate2

Falcon-11B support

Open
#1,737 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
4.7k
Forks
537
Avg merge
12h 12m
Merged PRs (30d)
4

Description

Hi
I converted Falcon-11b using ct2-transformers-converter but I get the following error when trying to use the model.

Ctranslate version: 4.1.0

Conversion command:

ct2-transformers-converter --model tiiuae/falcon-11b --output_dir falcon-11b-base-ct2 --quantization int8 --trust_remote_code

Using model:

>>> import ctranslate2
>>> from transformers import AutoTokenizer
>>> model = ctranslate2.Generator("falcon-11b-base-ct2", device='cpu')
>>> tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-11b")
>>> outputs = model.generate_batch([tokenizer.convert_ids_to_tokens(tokenizer.encode("Falcon 11b is a new LLM"))], sampling_topk=10, max_length=200, include_prompt_in_result=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: axis 2 has dimension 6144 but expected 4352
>>> 

I assume this is due to the new model being unsupported, will falcon 11b be supported by CT2?
Thanks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the ct2-transformers-converter command and the ctranslate2.Generator example from the issue with Falcon-11B. Trace the conversion and generation entry points to identify where the 6144-versus-4352 dimension mismatch occurs. Done means Falcon-11B converts and generates successfully with the reported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.