Cropping a segmented object from various product image

I'm currently developing a cropping tool that run on top of the SAM2.1 base Model weight checkpoint. Since this process are considered in a backend process, there can't be any manual adjustment such as manually dragging the ROI / bounding box, determine the object location in the image by coordinate, etc.)

What i'd like to do is listed below:
1. Remove the background
2. separately cropped the segmented object away from any other elements present in the image (product title text/ sub-text/and other decoration element)
3. return the cropped object as a result

The project takes input from product advertisement image that usually came up with promotional graphic assets [like this input image example](https://i.sstatic.net/f5jCVXT6.jpg).
I've tried to combine python rembg library as a preprocessing step before passing the image into the SAM Mask Generator, but it didn't do well especially when the text & other graphics doesn't look like a background at all. Is there any other way that i can try?

the expected output should only contain the segmented object(s), currently my model segment like this [Image](https://github.com/user-attachments/assets/a6b44765-49f2-4a52-b862-acd4fd23a0ca)

```python
image_pil = PIL.Image.open(image_path)
image_np = np.array(image_pil)

# Ensure 3 color channels
if image_np.shape[2] == 4:
image_np = image_np[:, :, :3]

total_area = image_np.shape[0] * image_np.shape[1]
print(f"Total area: {total_area}")

# Generate masks
mask_generator = SAM2AutomaticMaskGenerator(
model=self.sam2,
points_per_side=32,#64,
points_per_batch=64,#128,
pred_iou_thresh=0.7,
stability_score_thresh=0.92,
stability_score_offset=0.7,
crop_n_layers=1,
box_nms_thresh=0.7
)
masks = mask_generator.generate(image_np)
print(f"found {len(masks)} initial masks")

filtered_masks = []
for mask in masks:
mask_area = mask["area"]
area_percentage = mask_area / total_area

if self.min_area_percentage <= area_percentage <= self.max_area_percentage:
filtered_masks.append(mask)

masks = filtered_masks
masks.sort(key=lambda x: x["area"], reverse=True)

selected_masks = masks[:self.max_objects]

# Create output directory if not exists
os.makedirs(output_dir, exist_ok=True)

# Store extracted object paths
extracted_objects_paths = []

# Process masks
for mask_data in selected_masks:
# Get mask
mask = mask_data["segmentation"]

# Create image with alpha channel
rgba_image = np.zeros((image_np.shape[0], image_np.shape[1], 4), dtype=np.uint8)

# Fill RGB channels
for c in range(3):
rgba_image[:, :, c] = image_np[:, :, c]

# Fill alpha channel based on mask
rgba_image[:, :, 3] = (mask * 255).astype(np.uint8)

# Find bounding box of the object
rows = np.any(mask, axis=1)
cols = np.any(mask, axis=0)
ymin, ymax = np.where(rows)[0][[0, -1]]
xmin, xmax = np.where(cols)[0][[0, -1]]

# Crop image with bounding box
cropped_rgba = rgba_image[ymin : ymax + 1, xmin : xmax + 1]

# Convert to PIL Image
cropped_pil = PIL.Image.fromarray(cropped_rgba)

output_filename = f"{uuid.uuid4()}.png"
output_path = os.path.join(output_dir, output_filename)

# Save as PNG with transparency
cropped_pil.save(output_path, format="PNG")

# Store extracted object path
extracted_objects_paths.append(output_path)

return extracted_objects_paths
```