huggingface / huggingface/transformers.js

[Question] Any examples of processing video frames of a user uploaded video (specifically for depth estimation)?

Open
#417 3 comments 1 reaction 0 assignees View on GitHub
question
Dominant language
JavaScript
Stars
16.3k
Forks
1.2k
Avg merge
6d 2h
Merged PRs (30d)
6

Description

Hi there, I'm wondering if there are any examples of processing video frames of a user uploaded video? I'm specifically looking to run depth estimation on each frame of a short video, but any similar example would be useful.

If not, does this approach seem correct?
* Use one of the approaches described [here](https://stackoverflow.com/questions/32699721/javascript-extract-video-frames-reliably) to draw each frame of the video to a canvas
* Call `HTMLCanvasElement.toBlob()` on the canvas to get a `Blob`
* Pass N (10?) of those Blobs to a worker at a time
* For each of those Blobs call `const image = await RawImage.fromBlob(blob)` to get a `RawImage`
* Run depth estimation on the list of images with `await classifier([rawImage1, rawImage2, etc.])`

Thanks for any help!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.