What exactly is object pointer?

Hi, I would appreciate it if someone could shed some light on the `object pointer` feature introduced in SAM2.

As mentioned in the paper:
`
In addition to the spatial memory, we store a list of object pointers as lightweight vectors for high-level semantic information of the object to segment, based on mask decoder output tokens of each frame.
`

However, in the (inference) code, I just don't see the benefit of this particular token. Moreover, the tracking still seems to work if you disable it. I wonder if someone could share their insights on this.