OpenNMT / OpenNMT/CTranslate2

Whisper Model Has Inaccessible Logits/Log Probs

Open
#1,855 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
4.7k
Forks
536
Avg merge
12h 12m
Merged PRs (30d)
4

Description

Description:
Hello, I've been tinkering with CTranslate2 with whisper converted to ct2. Using model.generate with return_logits_vocab=True and return_scores=True I'm unable to access or print out the logits entirely. For example:

result = model.generate(..., return_logits_vocab=True, return_scores=True)
print(result[0].logits[0][0])
# Output: [cpu:0 float32 storage viewed as ]
print(result[0].logits[0][0].__array_interface__)
# Output: {'shape': (), 'typestr': '<f4', 'data': (0, False), 'version': 3}

This issue is reproducible with any input, even though the sequences are transcribed correctly.

Expected Behavior:
I would expect result[0].logits to contain accessible tensors with the logit values or log probs for each token generated.

Actual Behavior:
The logits appear to be empty or not fully accessible, as indicated by the output above.

This is super strange to me because result[0].scores[0] is completely accessible and prints out exactly as expected and I'd have to imagine that the scores attribute is just some mean or derivative of all of the log probs/logits for each token generated by CTranslate2. Which leads me to believe that this is some sort of bug rather than intentional behavior to hide the log probs.

Additional Details:

  • CTranslate2 Version: 4.5.0

  • I also tried the latest commit (6a3dc63db5608567bf554ca5dbbfdac1f7a2321f) which seemed related to the logits issues. While the commit added the logits property on print WhisperGenerationResult with print(result[0]), even there the issue persists with the same empty logits and everything else working:

    logits=[[
    [cpu:0 float32 storage viewed as ]], [
    [cpu:0 float32 storage viewed as ]], [
    [cpu:0 float32 storage viewed as ]], [
    [cpu:0 float32 storage viewed as ]], [
    [cpu:0 float32 storage viewed as ]]
    

Environment:
Python 3.11.11
MacOS Sequoia 15.2

Running cpu float32 inference on an M1 Pro Macbook.

Please let me know if you need any more information. If this is indeed a bug (and not intended behavior for the Whisper model), I'm happy to investigate further or contribute a fix.

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the Whisper path through the Python model.generate call with return_logits_vocab=True and return_scores=True, then inspect the WhisperGenerationResult logits exposed by that API. Done means result[0].logits contains accessible values for each generated token, while the existing scores behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.