Imageomics / Imageomics/pybioclip

Support prediction from pre-computed image embeddings

Open
#167 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
67
Forks
14
PR merge metrics
No merged PRs in 30d

Description

## Problem

Currently, the `predict()` methods on `TreeOfLifeClassifier`, `CustomLabelsClassifier`, and `CustomLabelsBinningClassifier` only accept raw images (file paths or PIL images). Internally, `predict()` always runs the full image encoding pipeline (`create_image_features()`) before computing classification probabilities.

This means that users who need **both** the image embeddings and the classification results must encode the image twice:

```python
classifier = TreeOfLifeClassifier(device='cuda')

# First pass: get embeddings for downstream use (e.g., similarity search)
features = classifier.create_image_features([classifier.ensure_rgb_image(img)])

# Second pass: predict re-encodes the same image internally
results = classifier.predict(images=img, rank=Rank.SPECIES)
```

The image encoding through the ViT backbone is by far the most expensive step in the pipeline. This doubles compute cost unnecessarily.

## Use cases

1. **Embedding + classification in one workflow**: Applications like [bioclip-image-search-lite](https://github.com/Imageomics/bioclip-image-search-lite) classify images and embed them for similarity search. With the current API, this requires encoding every image twice.
2. **Deferred classification**: A user pre-computes and stores embeddings (e.g., via the `bioclip embed` CLI), then later wants to classify them without needing the original images or GPU-heavy re-encoding.

## Proposed solution

Add an optional `image_features` parameter to the `predict()` methods. When provided, the method skips image encoding and computes classification probabilities directly from the supplied embeddings.

The method should validate the input embeddings before classification:
- Verify the tensor is 2D `(N, embedding_dim)`
- Check that `embedding_dim` matches the model's expected dimension (e.g., 768 for ViT-L/14)
- Normalize embeddings via L2 normalization before computing similarity, consistent with how BioCLIP 2 handles features in its forward pass

```python
# Encode once
features = classifier.create_image_features([classifier.ensure_rgb_image(img)])

# Predict from pre-computed embeddings — no re-encoding
results = classifier.predict(image_features=features, rank=Rank.SPECIES)
```

The change is fully backward-compatible — existing call signatures remain unchanged.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the predict() methods on TreeOfLifeClassifier, CustomLabelsClassifier, and CustomLabelsBinningClassifier, then compare them with create_image_features() and the bioclip embed CLI flow. Done means pre-computed embeddings can be classified with shape and dimension validation, normalization, and no repeated image encoding while existing image-based calls remain supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.