07th-mod / 07th-mod/higurashi-assembly

Computed Lipsync Improvements

未關閉
#89 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
C#
星號
12
分支
12
PR 合併指標
30 天內沒有已合併 PR

描述

I noticed that the new computed lipsync sometimes has the mouth completely closed near the end of a sentence, if the character is speaking softly.

This is probably due to the current algorithm using fixed thresholds with no hysteresis, so if a character is speaking just below the threshold (currently 30% off max volume), the mouth will always stay closed.

Another related issue is that sometimes a character's mouth will be 'stationary'.

I can think of a couple things to try to fix this:

1. Try to detect changes in volume + also use fixed thresholds to determine which mouth to use
a. If the long term average volume is say, less than 50%, always use fully closed or half-open mouth (talking softly)
b. If the long term average volume is greater than 50%, always use the half-open or fully open mouth (talking loudly)
c. If the volume suddenly increases, use the more open mouth position, otherwise, use the more closed position
d. Only hold the mouth in the fully closed position if the volume becomes very low (another threshold? like 10% or 5%)
2. Use fixed thresholds but:
a. Add hysteresis only when going from half-closed to fully closed, to prevent character's mouths closing too early.
b. If a character remains in one mouth position too long, switch to another position even if the average volume hasn't changed

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

在程式碼庫中尋找唇形同步演算法,很可能在處理音訊或角色動畫的檔案中。該問題描述了閾值和遲滯;找到將音量與30%閾值進行比較的位置。透過在角色輕聲說話的範例音訊上執行唇形同步來測試更改。驗證嘴巴不會錯誤地保持閉合狀態且不會變得靜止。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
csharp
領域
audio-video-rtc
Issue 類型
缺陷
難度
3/5
預估耗時
1-2 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。