Implement feature extraction module for verification
- Dominant language
- Go
- Stars
- 283
- Forks
- 72
- Avg merge
- 27m
- Merged PRs (30d)
- 1
Description
### Abstract
In our decentralized video network, it is important part to detect tamper videos to prevent the myriad of malicious attacks seeking to misinform, cheat to digital multimedia audience.
This is a proposal for implementing verification module in lpms engine.
### Motivation
We already developed verifier that run on broadcaster, this verifier has fairly good accuracy for detecting tampered videos.
But we still need outsourcing verifier to reduce the calculation cost in broadcaster and we can introduced TTP(Truste
d Third Part) similar to [Zcash](https://en.wikipedia.org/wiki/Zcash) and can be used to implement the verification workflow.
To implement this, first, the feature values to be used for verification must be calculated during transcoding.
### Proposed Solution
Feature extraction module should proceed simultaneously with transcoding and should be real
time and we also would want the feature diff b/w the source & rendition(s) as well as the source and the extra video.
From this view point, I am going to integrate parallel independent decoding of extra video and
ffmpeg api for calculation feature matrix.
The diagram below shows the work flow of feature extraction module in lpms engine.
In this diagram, if the number of transcoding profiles is two, we would get three different feature matrixes.
One is feature diff between the source and extra, others feature diffs between the source and renditions.

- GO Level
As we can see in above diagram, the parameters for transcoding(+verification) will be two video local url.
Therefore, we should add a function with a string array as parameters in following URL.
https://github.com/livepeer/lpms/blob/d5c85d86b206bddb63859921eca89f5c1f267b1e/transcoder/ffmpeg_segment_transcoder.go#L24
The return values are renditions byte array and the final diff score between original and extra video in the json form.
func (t *FFMpegSegmentTranscoder) Transcode2(fname []string) ([][]byte, diffscore []string, error) {
… … …
}
- C Level
As with Go level, we have to add a function which has two local urls for videos(original & extra) as inputs
https://github.com/livepeer/lpms/blob/d5c85d86b206bddb63859921eca89f5c1f267b1e/ffmpeg/lpms_ffmpeg.c#L1537
int lpms_transcode2(input_params *inp, int nb_inputs, output_params *params,
output_results *results, int nb_outputs, output_results *decoded_results)
struct transcode_thread {
int initialized;
struct input_ctx ictx;
struct output_ctx outputs[MAX_OUTPUT_SIZE];
int nb_outputs;
... ... ...
AVFrame *list_frame_original
AVFrame *list_frame_renditions
};
And a list will be added into the " transcode_thread" or “input_ctx” structure to captures frames based on random frame indices. When transcoding, the frames of originals and renditions is stored into this list at our source [here ](https://github.com/livepeer/lpms/blob/d5c85d86b206bddb63859921eca89f5c1f267b1e/ffmpeg/lpms_ffmpeg.c#L1293)and and [here](https://github.com/livepeer/lpms/blob/d5c85d86b206bddb63859921eca89f5c1f267b1e/ffmpeg/lpms_ffmpeg.c#L159)
Meanwhile, a thread is used to perform decoding and frame capture of a extra video in parallel.
Finally, call lvpdiff api of ffmpeg to calculate the frame different scores and generate the return value.
I will refer to [here ](https://github.com/livepeer/lpms/blob/d5c85d86b206bddb63859921eca89f5c1f267b1e/ffmpeg/lpms_ffmpeg.c#L1509)and [here ](https://github.com/oscar-davids/FFmpeg/blob/101296fd70f0df25c24bb4c7481536da91910f95/libavfilter/avfilter.h#L1167)in our code.
Ideally this function with two inputs will be take 0.2~0.4 seconds more than original transcoding function with one input.
### Testing and Considerations
As like original transcoding function with one file input, in the "ffmpeg_test.go" should be added test function related to two inputs.
### References
https://www.notion.so/livepeer/Real-Time-Verification-Thoughts-0a0ad16546a54dc3b77589f01f2bc333
https://github.com/livepeer/verification-classifier/tree/master/scripts
https://stackoverflow.com/questions/37353250/may-i-use-opencv-in-a-customized-ffmpeg-filter
https://en.wikipedia.org/wiki/Zcash
https://en.wikipedia.org/wiki/Zero-knowledge_proof
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading transcoder/ffmpeg_segment_transcoder.go around the linked Transcode entry point and ffmpeg/lpms_ffmpeg.c around the referenced transcode and frame-processing locations. Review ffmpeg_test.go for the existing single-input behavior; done would require a tested two-input transcoding path that returns renditions and verification diff results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, go
- Domain
- audio-video-rtc, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100