modelscope / modelscope/FunASR
[Bug / Performance] spk_model spectral clustering saturates every CPU core by default
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 20.4k
- Forks
- 2k
- Avg merge
- 4h 55m
- Merged PRs (30d)
- 169
Description
Summary
Speaker diarization (spk_model="cam++") pins every CPU core during the
clustering tail of a request. No preset_spk_num is needed -- this is the
default path, and it happens even for short audio.
Environment
- FunASR 1.2.7 (also reproduced on current
main) - 64-core host, Tesla T4
AutoModel(model=..., vad_model="fsmn-vad", punc_model="ct-punc", spk_model="cam++", device="cuda")- 35-minute recording -> N=357 segments
Symptom
Container CPU over one request:
t=44s cpu=100.79% <- ASR on GPU
t=48s cpu=5767.99% <- 57.7 of 64 cores, GPU utilisation back to 0
t=56s cpu=5767.21%
t=60s cpu=1.79% <- done
All of it lands in one ClusterBackend pass at the very end of the request.
Root cause
SpectralCluster.get_spec_embs calls a full dense scipy.linalg.eigh on
the N x N affinity Laplacian, then uses almost none of the result:
- the speaker count reads only the gaps among the first
max_num_spks + 1
(i.e. 16) eigenvalues; - the embedding keeps only the first
num_of_spkeigenvectors.
Two independent problems follow:
- The full spectrum is O(N^3) work that is thrown away.
scipy.linalg.eighgoes through BLAS, whose default thread count is one
per core. At a few hundred rows the parallel drivers spend more time
synchronizing than computing.
Measured with threadpoolctl.threadpool_info(), the process had:
openblas (numpy) num_threads=64
openblas (scipy) num_threads=64
libgomp (scikit-learn) num_threads=64
libgomp (torch) num_threads=4 <- only torch is capped
AutoModel's ncpu -> torch.set_num_threads only reaches the torch pool,
so the clustering call is left unbounded.
Instrumenting the pipeline (wall / CPU in core-seconds, N=357):
ClusterBackend.forward 6.553s / 327.04 core-s
get_sim_mat 0.063s / 2.61
p_pruning 0.148s / 5.92
get_spec_embs (eigh) 6.190s / 312.04 <-- 50 cores average
cluster_embs (k_means) 0.113s / 6.42
Microbenchmark
eigh on the same input, varying BLAS threads:
threads n=357 wall n=357 CPU n=1200 wall n=1200 CPU
1 0.030s 0.03 0.746s 0.74
4 0.027s 0.10 0.471s 1.31
32 0.037s 1.06 0.500s 9.93
64 0.973s 57.01 3.701s 205.72
Wall-clock is essentially flat from 1 to 32 threads while CPU scales linearly
-- the extra threads are pure overhead, and at 64 they make it slower.
Suggested fix
Both changes are independent; together they are ~17x faster and ~950x cheaper
in CPU on n=1200:
# only the leading eigenpairs are ever used
n_eig = max_num_spks + 1
if k_oracle is not None:
n_eig = max(n_eig, int(k_oracle))
n_eig = min(n_eig, L.shape[0])
lambdas, eig_vecs = scipy.linalg.eigh(L, subset_by_index=[0, n_eig - 1])
# and cap BLAS, which defaults to one thread per core
with threadpoolctl.threadpool_limits(limits=1, user_api="blas"):
...
Combined, on the same hosts:
n=1200 full + 64 threads 4.278s / 237.59 core-s (before)
subset + 1 thread 0.246s / 0.25 core-s
End-to-end the ClusterBackend pass goes from 327 -> 8.7 core-seconds and
6.55s -> 0.99s, with byte-identical speaker labels (761 sentences,
{0:322, 1:156, 2:144, 3:139}). A sweep over n=3..900 and
k_oracle in {None, 2, 20, 40} reproduces the original speaker counts and
eigenvector subspaces exactly, including the n < max_num_spks + 1 edge case.
A PR is linked below.
Relationship to #3514
#3514 covers the large-N + preset_spk_num path, where the fix routes to
kmeans_cluster. That fix is correct but does not cover this report:
- here
X.shape[0] = 357 < 2048, soClusterBackend.forwardtakes
if X.shape[0] < 2048: spectral_cluster(X, k)-- a different branch; preset_spk_numis not set at all in the repro above.
Neither path had any BLAS thread control, before or after #3514, so the
core-saturation half of this is present in all versions up to and including
1.4.15 (verified by diffing cluster_backend.py across 1.2.7, 1.2.9, 1.3.0,
1.3.10, 1.3.20, 1.3.30, 1.4.7, 1.4.15).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in cluster_backend.py at ClusterBackend.forward, then follow the spectral_cluster path into SpectralCluster.get_spec_embs. Reproduce the 357-segment case and inspect the existing scipy.linalg.eigh call and thread pools. Done means limiting eigenpairs and BLAS threads without changing speaker counts or eigenvector results, including the stated edge cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch, scikit-learn
- Domain
- backend, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100