Reading reduced number of frames to improve performance

Great Project. I am just getting started building a simple program to detect large differences between frames. Reading every video frame for large files takes quite a while so I am trying to just read a couple frames per second to improve the speed. However, it is slower when I try to choose which frames to read.

Here is the minimal code I am using in a simple loop reading only a couple frames per second with the onFrame callback doing no work:
readFrameByNumber(frameNumber, onFrame) {
var time = Date.now();
this._cvVideo.set(cv.CAP_PROP_POS_FRAMES, frameNumber-1);
console.log('set ' + (Date.now() - time) + ' ms');
time = Date.now();
this._cvVideo.readAsync((err, frame) => {
console.log('readAsync ' + (Date.now() - time) + ' ms');
onFrame();
});
}

And here is the timing metrics in the console:
file: mp4 (from a samsung phone camera)
fps: 29.947830816905125
frame count: 3553
duration: 118.63964444444444 secs
resolution: 1920, 1080
set 41 ms
readAsync 44 ms
set 120 ms
readAsync 8 ms
set 166 ms
readAsync 8 ms
set 126 ms
readAsync 8 ms
set 119 ms
readAsync 8 ms
set 144 ms
readAsync 8 ms
set 74 ms
readAsync 8 ms
set 73 ms
readAsync 9 ms
set 139 ms
readAsync 10 ms
set 70 ms
readAsync 8 ms
set 127 ms
readAsync 8 ms

Is there a reason the set this._cvVideo.set(cv.CAP_PROP_POS_FRAMES, frameNumber-1); command is so slow? When I read every frame of the file it is about 8ms per read, I was hoping to improve that by reading less frames evenly spaced out in the video. Is there another way to read the frames faster or a better way to read less frames without incurring the cost of seeking to each frame position?

Please let me know if you have any ideas or if I am just approaching this incorrectly.

Thanks

OpenCV version: 5.5.0
NWJS: 0.42.5
With OpenCV-contrib: no
Windows 10