Tencent-RTC / Tencent-RTC/TRTC_Web

[Feature] FaceDetection plugin should support real inference FPS and input resolution control

Open
#159 2 comments 0 reactions 2 assignees View on GitHub

@Rychou is already working on this.

Since Jul 3, 2026.

enhancement
Dominant language
JavaScript
Stars
160
Forks
72
Avg merge
6m
Merged PRs (30d)
2

Description

Is your feature request related to a problem? Please describe.

The current FaceDetection plugin appears to run face detector inference on every video frame, while detectionInterval only throttles state handling / callbacks. This makes it difficult to use FaceDetection in low-end devices or TRTC applications that need very low overhead face-presence detection.

From the published trtc-sdk-v5@5.18.2 package:

  • plugins/video-effect/face-detection/face-detection.esm.js: detectFace() calls getCurrentStatus() before checking _detectionInterval.
  • getCurrentStatus() calls this._visionTaskRegistry.getResult(this._faceDetectorHash).detections and then resetHashResults().
  • assets/mediapipe/vision.js: FaceDetector is mapped to instance.detectForVideo.bind(instance).
  • VisionTaskRegistry.reasoning() calls the reasoning function with video and performance.now().
  • detectFace() schedules itself again with video.requestVideoFrameCallback(this.detectFace.bind(this)).

So with a 30 FPS camera stream, detectForVideo may run close to 30 times per second even if detectionInterval is set to a much larger value.

Describe the solution you'd like

Please add options to the FaceDetection plugin to control actual inference workload:

  1. inferenceFps or fps: limit how often detectForVideo is called, for example 1-5 FPS for lightweight face-presence checks.
  2. inputResolution: allow downsampling before inference, for example { width: 96, height: 72 }, { width: 160, height: 120 }, or similar.
  3. Keep detectionInterval for callback/state debounce, or clarify its documentation if it is not intended to throttle model inference.

A possible implementation would be:

  • Move the time check before getCurrentStatus(), so expensive inference is skipped until the next inference window.
  • Optionally use a canvas / OffscreenCanvas as the FaceDetector input when inputResolution is provided.
  • Preserve the current behavior as the default to avoid breaking existing users.

Describe alternatives you've considered

Applications can implement a custom MediaPipe FaceDetector pipeline with low-resolution sampling and low FPS, but this duplicates functionality already provided by the TRTC plugin and requires separate handling of camera/video resources.

Additional context

This is useful for applications that already use TRTC and only need a simple hasFace: boolean signal, especially when running alongside audio/video publishing on low-end clients. In such cases, 1-2 FPS and a small input resolution are often enough and can significantly reduce CPU usage.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.