Imageomics / Imageomics/pybioclip
Support prediction from pre-computed image embeddings
- Dominant language
- Python
- Stars
- 67
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Currently, the `predict()` methods on `TreeOfLifeClassifier`, `CustomLabelsClassifier`, and `CustomLabelsBinningClassifier` only accept raw images (file paths or PIL images). Internally, `predict()` always runs the full image encoding pipeline (`create_image_features()`) before computing classification probabilities.
This means that users who need **both** the image embeddings and the classification results must encode the image twice:
```python
classifier = TreeOfLifeClassifier(device='cuda')
# First pass: get embeddings for downstream use (e.g., similarity search)
features = classifier.create_image_features([classifier.ensure_rgb_image(img)])
# Second pass: predict re-encodes the same image internally
results = classifier.predict(images=img, rank=Rank.SPECIES)
```
The image encoding through the ViT backbone is by far the most expensive step in the pipeline. This doubles compute cost unnecessarily.
## Use cases
1. **Embedding + classification in one workflow**: Applications like [bioclip-image-search-lite](https://github.com/Imageomics/bioclip-image-search-lite) classify images and embed them for similarity search. With the current API, this requires encoding every image twice.
2. **Deferred classification**: A user pre-computes and stores embeddings (e.g., via the `bioclip embed` CLI), then later wants to classify them without needing the original images or GPU-heavy re-encoding.
## Proposed solution
Add an optional `image_features` parameter to the `predict()` methods. When provided, the method skips image encoding and computes classification probabilities directly from the supplied embeddings.
The method should validate the input embeddings before classification:
- Verify the tensor is 2D `(N, embedding_dim)`
- Check that `embedding_dim` matches the model's expected dimension (e.g., 768 for ViT-L/14)
- Normalize embeddings via L2 normalization before computing similarity, consistent with how BioCLIP 2 handles features in its forward pass
```python
# Encode once
features = classifier.create_image_features([classifier.ensure_rgb_image(img)])
# Predict from pre-computed embeddings — no re-encoding
results = classifier.predict(image_features=features, rank=Rank.SPECIES)
```
The change is fully backward-compatible — existing call signatures remain unchanged.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the predict() methods on TreeOfLifeClassifier, CustomLabelsClassifier, and CustomLabelsBinningClassifier, then compare them with create_image_features() and the bioclip embed CLI flow. Done means pre-computed embeddings can be classified with shape and dimension validation, normalization, and no repeated image encoding while existing image-based calls remain supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100