alphacep / alphacep/vosk-api

Questions/change request proposal about decoding output format (result, partialResult, finalResult)

Offen
#608 5 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Jupyter Notebook
Sterne
15.1k
Forks
1.8k
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Hi Nicolay

following you comment on closed issue: https://github.com/alphacep/vosk-api/issues/590#issuecomment-863065813, you stated that:
> 1. AcceptWaveform returns true when silence is detected and you can retrieve the result with Result().
> 2. If silence is not detected you can retrieve PartialResult() only.
> 3. FinalResult means the stream is ended, you flush the buffers and retrieve remaining result.

Now, I tested above methods processing a speech file containing some sentences separated by "silences":

`one two three` silence `four five six seven eight` silence `nine zero one two three stop`

So the audio contains "three" different sentences (considering that the silence / end of audio terminate a sentence).

- `one two three`
- `four five six seven eight`
- `nine zero one two three stop`

Here the audio: https://github.com/solyarisoftware/voskJs/blob/master/audio/sentencesWithSilences.wav

If I run my voskjs (nodejs cli wrapper to vosk-api) I get:
```
$ voskjs --audio=audio/sentencesWithSilences.wav --model=models/vosk-model-small-en-us-0.15

model directory : models/vosk-model-small-en-us-0.15
speech file name : audio/sentencesWithSilences.wav
grammar : not specified. Default: NO
sample rate : not specified. Default: 16000
max alternatives : undefined
text only / JSON : JSON
Vosk debug level : -1

load model latency : 285ms

partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: 'one' }
partialResult { partial: 'one' }
partialResult { partial: 'one two' }
partialResult { partial: 'one two' }
partialResult { partial: 'one two three' }
endOfSpeechResult {
result: [
{ conf: 1, end: 1.95, start: 1.41, word: 'one' },
{ conf: 1, end: 2.7, start: 2.28, word: 'two' },
{ conf: 1, end: 3.96, start: 3.45, word: 'three' }
],
text: 'one two three'
}
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: 'for' }
partialResult { partial: 'four five six' }
partialResult { partial: 'four five six' }
partialResult { partial: 'four five six seven' }
partialResult { partial: 'four five six seven' }
partialResult { partial: 'four five six seven eight' }
partialResult { partial: 'four five six seven eight' }
partialResult { partial: 'four five six seven eight' }
endOfSpeechResult {
result: [
{ conf: 1, end: 7.44, start: 6.78, word: 'four' },
{ conf: 1, end: 8.28, start: 7.68, word: 'five' },
{ conf: 1, end: 9.18, start: 8.37, word: 'six' },
{ conf: 1, end: 9.96, start: 9.24, word: 'seven' },
{ conf: 1, end: 10.8, start: 10.17, word: 'eight' }
],
text: 'four five six seven eight'
}
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: 'nine' }
partialResult { partial: 'nine zero' }
partialResult { partial: 'nine zero' }
partialResult { partial: 'nine zero' }
partialResult { partial: 'nine zero one two' }
partialResult { partial: 'nine zero one two' }
partialResult { partial: 'nine zero one two three' }
partialResult { partial: 'nine zero one two three' }
partialResult { partial: 'nine zero one two three stop' }
partialResult { partial: 'nine zero one two three stop' }
endOfSpeechResult {
result: [
{ conf: 1, end: 13.74, start: 13.05, word: 'nine' },
{ conf: 1, end: 14.64, start: 13.95, word: 'zero' },
{ conf: 1, end: 15.42, start: 14.79, word: 'one' },
{ conf: 1, end: 16.17, start: 15.69, word: 'two' },
{ conf: 1, end: 16.98, start: 16.47, word: 'three' },
{ conf: 1, end: 17.85, start: 17.28, word: 'stop' }
],
text: 'nine zero one two three stop'
}
partialResult { partial: '' }
finalResult { text: '' }

transcript latency : 1880ms
```

Vosk correctly transcript all sentences, detecting the end_of_speech, but I have some related questions and proposals:

### Question 1
How is calculated the `end_of_speech`?
I searched in un the C code and It seems to me that's related to `Endpointdetected` function:
```
$ rg EndpointDetected
src/kaldi_recognizer.cc
307: if (decoder_->EndpointDetected(model_->endpoint_config_)) {
```
But I didn't find how is defined/how it works EndpontDetected. Anyway my question is about how is calculated the "silence". Maybe the calculation is about the elapsed time (in milliseconds, call it `SilenceDuration`) when volume (RMS) is under a threshold value (call it `TresholdVolume`) ?

If so, can I configure/set `TresholdVolume` and/or `SilenceDuration` in the vosk-api ? It is possible to allow vock-api user to configure/tune the silence attributes?

### Question 2

Consider the first sentence `one two three` partial results:
```
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: '' }
partialResult { partial: 'one' }
partialResult { partial: 'one' }
partialResult { partial: 'one two' }
partialResult { partial: 'one two' }
partialResult { partial: 'one two three' }
```

Why I get 5 times a void string partial result?
Ok, I can discard these void strings in my application, but why these void results are emitted? There is something underneath I do not see? If no, my proposal is to do not emit these void strings.

### Question 3

As far as I understand the `finalResult` attribute run as expected (collecting the full transcript of the audio) just if the audio do not contains multi sentences (sentences separated by silences) if the audio do not contains end_of_speeches.

Otherwise the finalResult does not contain nothing (that result in the reported example). That seems to me a possible functional "bug". I would expect instead/I'd propose the finalResult to be a list of sentences (partialresult); something like this:

```
[
{
words: [
{ conf: 1, end: 1.95, start: 1.41, word: 'one' },
{ conf: 1, end: 2.7, start: 2.28, word: 'two' },
{ conf: 1, end: 3.96, start: 3.45, word: 'three' }
],
text: 'one two three'
},

{
words: [
{ conf: 1, end: 7.44, start: 6.78, word: 'four' },
{ conf: 1, end: 8.28, start: 7.68, word: 'five' },
{ conf: 1, end: 9.18, start: 8.37, word: 'six' },
{ conf: 1, end: 9.96, start: 9.24, word: 'seven' },
{ conf: 1, end: 10.8, start: 10.17, word: 'eight' }
],
text: 'four five six seven eight'
},

{
words: [
{ conf: 1, end: 13.74, start: 13.05, word: 'nine' },
{ conf: 1, end: 14.64, start: 13.95, word: 'zero' },
{ conf: 1, end: 15.42, start: 14.79, word: 'one' },
{ conf: 1, end: 16.17, start: 15.69, word: 'two' },
{ conf: 1, end: 16.98, start: 16.47, word: 'three' },
{ conf: 1, end: 17.85, start: 17.28, word: 'stop' }
],
text: 'nine zero one two three stop'
}
]
```

What do you think?
Thanks for your patience!
Giorgio

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.