[Question] Using VisualizerWithVertexSelection to edit point cloud

I want to select points in a rectangular region and remove them from the point cloud. I am using VisualizerWithVertexSelection to select the points but haven't been successful so far in removing the points. I tried calling PointCloud.select_by_index() to get a new point cloud without the point indices selected. That works but then updating the geometry of the visualizer with the new point cloud does nothing. Is there a way to accomplish what I want?