CLI predict reloads model on every invocation, consider persistent session

Workshop observation

During the workshop, I have two attendees noting that bioclip predict felt slow when running more than a handful of predictions (predicting one image at a time). After looking at the source, I believe this is primarily because the CLI loads the full model, applies torch.compile(), and downloads the text embedding cache on every invocation, while none of which is cached across calls.

Flagging this as a non-priority CLI UX observation rather than a bug, since the current behavior works correctly and is simple to reason about. But the model loading dominates wall time, it meaningfully affects the interactive experience.

What happens on each bioclip predict call

In src/bioclip/__main__.py, the predict command instantiates a fresh classifier every time. For TreeOfLifeClassifier, this triggers a sequence of expensive setup in src/bioclip/predict.py:

• oc.create_model_from_pretrained(...) loads ~2.5 GB of weights from the HF cache
• torch.compile(model.to(self.device)) triggers JIT compilation overhead on first forward pass (only works on Linux OS, bypass fallback for Windows)
• TreeOfLifeClassifier downloads/loads txt_emb_species.npy (the TreeOfLife-200M text embedding table, BioCLIP 2 default) and its metadata JSON

Actual prediction is fast relative to this setup, so for users running predict repeatedly the startup cost dominates.

Proposed direction

A persistent background session would eliminate the per-call cost. I'm implementing something similar for BioCLIP Image Search Lite CLI development, loading the FAISS index, model weights, duckDB in-memory in a background session. Roughly:

• bioclip serve starts a local daemon that loads the classifier once
• bioclip predict image.jpg detects the running daemon and routes the request through it; falls back to the current in-process behavior if no daemon is running
• bioclip stop / bioclip status for lifecycle management
• An idle timeout so the daemon auto-shuts down if unused

Start in src/bioclip/main.py at the predict command, then read classifier setup in src/bioclip/predict.py, especially TreeOfLifeClassifier. Compare the current per-invocation loading of model weights, torch.compile(), and text embeddings with the proposed persistent-session flow. Done should include reusable loaded state for repeated predictions, lifecycle handling, and fallback behavior when no session is running.