07th-mod / 07th-mod/higurashi-assembly

Computed Lipsync Improvements

オープン
#89 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
enhancement
主要言語
C#
スター
12
フォーク
12
PR マージ指標
30日以内にマージされた PR はありません

説明

I noticed that the new computed lipsync sometimes has the mouth completely closed near the end of a sentence, if the character is speaking softly.

This is probably due to the current algorithm using fixed thresholds with no hysteresis, so if a character is speaking just below the threshold (currently 30% off max volume), the mouth will always stay closed.

Another related issue is that sometimes a character's mouth will be 'stationary'.

I can think of a couple things to try to fix this:

1. Try to detect changes in volume + also use fixed thresholds to determine which mouth to use
a. If the long term average volume is say, less than 50%, always use fully closed or half-open mouth (talking softly)
b. If the long term average volume is greater than 50%, always use the half-open or fully open mouth (talking loudly)
c. If the volume suddenly increases, use the more open mouth position, otherwise, use the more closed position
d. Only hold the mouth in the fully closed position if the volume becomes very low (another threshold? like 10% or 5%)
2. Use fixed thresholds but:
a. Add hysteresis only when going from half-closed to fully closed, to prevent character's mouths closing too early.
b. If a character remains in one mouth position too long, switch to another position even if the average volume hasn't changed

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

Look for the lipsync algorithm in the codebase, likely in a file handling audio processing or character animation. The issue describes thresholds and hysteresis; find where volume is compared to the 30% threshold. Test changes by running the lipsync on sample audio where a character speaks softly. Verify the mouth doesn't stay closed incorrectly and doesn't become stationary.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
csharp
領域
audio-video-rtc
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
45/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。