rasterize messes up label to shape mapping when rastering only a subset of shapes

Describe the bug
I ran in an issue when rasterizing shapes to labels. The mapping between labels and shapes seems to be messed up, when rasterize when the coordinate range (min_coordinate, max_coordinate) does not include all shapes.

To Reproduce
gist with a MWE

Expected behavior
Changing max_coordinate should keep label to shape mapping intact.

Screenshots
Just showing some screenshots here, fo the full code, check out the gist above.

Blob data with labeled shapes:

sdata.pl.render_shapes('blobs_polygons', color='position').pl.show()

After rasterizing them and linking the table, I plot them as labels, and this works (max_coordinate for x an y are 512)

However, if I only raster part of the image, the label_index_to_category attribute of the returned label element seems to be messed up. Here, I decresed the max_coordinate for x to 300. In the plot, you can only see part of the shapes, but suddenly, the labels appear mixed:

Desktop (optional):

• OS: macOS
• spatialdata version: '0.4.0'

Start with the rasterize call from the linked minimum working example, focusing on min_coordinate and max_coordinate and the returned label element's label_index_to_category attribute. Compare full-image rasterization with the x-limited case and trace how shapes are filtered and labels are mapped. Done means changing max_coordinate preserves the label-to-shape mapping while rasterizing only the selected range.