Imageomics / Imageomics/FuncaPalooza-2025

BioCLIP v2 orthogonal dims

Open
#9 2 comments 0 reactions 0 assignees View on GitHub
project idea
Dominant language
No language data
Stars
5
Forks
0
PR merge metrics
No merged PRs in 30d

Description

I personally would like to play around with the new and improved [BioCLIPv2](https://arxiv.org/pdf/2505.23883) embedding space. Specifically, if we can use the emergent properties of the embeddings for 0 shot or few shot classifiers or trait quantification.
From the CS side this is two ideas:
- Few-shot Classifier Ablations: Can we define actionable recommendations for creating "biologist ready" image classifiers for a "beyond species ID" task from BioCLIP 2 embeddings? Curious of data needs for different deployments, normalization, augmentation etc. Could apply on a novel task for understanding in a new biological context, and capture an interesting pattern across a different gradient. One idea could be for plants that have highly variable growth forms, and if we can quantify patterns in selection of growth form across large scales using inat or other image sources.
- Fine-grain explainability of orthogonal embedding dimensions for trait discovery or localization. [FinerCam](https://arxiv.org/pdf/2501.11309) can localize in-image features that cause class distinctions between two images. Is there a method we can develop that can localize image features that cause stratification along an emergent embedding axis other than species ID? For instance if a juvenile adult axis exists in the embedding space, can we show what image features cause separation of two samples along that access?

From the Bio side, I am open to whatever, but if it were up to me would like to work with phenology data. Can we create species agnostic classifiers of phenophase that could potentially be used to detect structures that inform that prediction?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked BioCLIPv2 and FinerCam papers, then narrow the proposal to either few-shot classifier ablations or localization of embedding axes. The issue does not identify repository files, tests, or an implementation entry point; done would require a defined experiment, biological task, evaluation plan, and documented results.

Written by the indexing model from the issue text.

Assessment

Tech stack
machine-learning
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.