Problem selecting pixels in the first frame

I am studying the segmentation of specific objects in the video. When selecting pixels, if the pixels of the first frame are not used, can I still segment all the frames in the entire video (assuming that the fifth frame of the image is used. If pixel points are selected, can the previous frames still be segmented? )