Cannot get correct translation results (NLLB-200-distilled-600M)
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 536
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
Hello, I am trying to use CTranslate2 to call the NLLB model, the program runs slowly and always returns rubbish results.
I compiled CTranslate2 and sentencepiece by Visual Studio 2022. Here are my codes:
#include <ctranslate2/translator.h>
#include <sentencepiece_processor.h>
#include <iostream>
#include <vector>
#include <string>
int main() {
std::string model_path = "F:/temp_work/LLM-models/nllb-200-distilled-600M-ct2-int8";
std::string sp_model_path = model_path + "/sentencepiece.bpe.model";
// Load the tokenize model
sentencepiece::SentencePieceProcessor processor;
const auto status = processor.Load(sp_model_path);
if (!status.ok()) {
std::cerr << status.ToString() << std::endl;
return -1;
}
// load the NLLB model
ctranslate2::Translator translator( model_path, ctranslate2::Device::CPU);
std::string input_text = u8"抽象代数重要的研究对象有:群、环和域。";
std::cout << input_text << std::endl;
// encode the string to token string
// You don't need to set what the language is.
std::vector<std::string> encoded_input_text;
processor.Encode(input_text, &encoded_input_text);
// encoded_input_text.insert(encoded_input_text.begin(), "zho_Hans");
//The input of the translation model should be a batch.
std::vector<std::vector <std::string>> input_batch = { encoded_input_text };
ctranslate2::TranslationOptions options;
options.max_decoding_length = 1024;
options.max_input_length = 512;
options.sampling_temperature = 0.7;
options.no_repeat_ngram_size = 0;
std::vector<std::vector<std::string>> target_prefix = { {"eng_Latn"} };
std::vector<ctranslate2::TranslationResult> results = translator.translate_batch(
input_batch,
target_prefix,
options);
std::vector<std::string> output = results[0].output();
std::string output_text;
processor.Decode(output, &output_text);
std::cout << "Translation: " << output_text << std::endl;
return 0;
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository file or test is identified; begin with the provided C++ reproduction and review the NLLB model input and output-token handling. Check the encoded input, language-token setup, target prefix, and decoded result, then compare CPU behavior and timing with the project's documented C++ usage. Done means a reproducible example produces a correct translation at an expected speed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100