GCWing / GCWing/OpenBitFun

[Optimize]: Resolution budgets, crop/bbox coordinate tools, and image-workflow guidance for vision tools

Ouverte
#2,248 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
Rust
Étoiles
2.2k
Forks
229
Merge moyen
2 h 46 min
PR mergées (30 j)
577

Description

## Summary

Upgrade BitFun's built-in vision tools with four capabilities borrowed from Qwen-MM-Plugins' design:

1. **Resolution budget presets** for `analyze_image` / `view_image` (`small` / `normal` / `large`)
2. **Coordinate closed loop**: new `crop_image` and `draw_bbox` tools using 0-1000 normalized coordinates
3. **Small-image upscaling** to a minimum pixel floor (capped at 2x linear) so tiny crops stay legible for VLMs
4. **Built-in decision skill** (`image-workflow`) teaching the agent when to view vs analyze vs crop, and how to pick a budget

## Background

The current vision path is two fixed-limit tools plus a batch pre-analysis path:

- `analyze_image` sends the image + prompt to the configured image-understanding model. Every call pays full image tokens, and the only size control is the hard provider cap (1MB tool cap / provider dimension limits). There is no way for the agent to trade detail against cost.
- `view_image` attaches an image to the primary model context, also with no resolution control.
- `optimize_image_with_size_limit` never upscales, so a tiny crop (e.g. a 64x64 region) is sent as-is and OCR/detail suffers.
- The only coordinate handling is a prose `coordinate_note` warning the model that positions in the analysis are estimates. There is no tool to actually crop or annotate a static image, so the estimate cannot be turned into an action loop.
- Tool descriptions are one-liners; the model has no guidance on which tool to use when.

Qwen-MM-Plugins solves these with token-budget-driven resolution presets (256/1024/2048 visual tokens), a `crop`/`draw_bbox` pair that consumes grounding output in the same 0-1000 normalized coordinate system, an upscale floor, and declarative SKILL.md decision documents.

## Proposed changes

### 1. Budget presets (optional parameter, legacy behavior unchanged)

- Add `ImageBudget` (small/normal/large) mapping to pixel targets via token budgets (256/1024/2048 x 32^2 = ~512^2 / ~1024^2 / ~1448^2), clamped by the provider `ImageLimits` ceiling.
- Add an optional `budget` parameter to `analyze_image` and `view_image`. When omitted, the existing path runs unchanged (no behavior change for existing callers/configs).
- Fix the `resize_note` wording which currently hard-codes "downscaled" and would misreport upscaling.

### 2. Coordinate closed loop: `crop_image` + `draw_bbox`

- `crop_image(path, box[x1,y1,x2,y2 in 0-1000], output_path?)`: validates and clamps the box, saves the cropped region next to the source by default, and returns pixel-mapped coordinates. Attaches a preview when the primary model supports multimodal tool output; degrades to a text summary otherwise.
- `draw_bbox(path, boxes[{bbox,label?}], output_path?)`: draws rectangles with an adaptive line width and a fixed palette. v1 draws boxes only (no text labels, no new dependencies).
- Both support remote workspace paths (read + write through workspace filesystem services).
- Closed loop: the model reports a region in normalized coordinates, `crop_image` cuts it, then `analyze_image(crop_path, budget="large")` re-inspects the region at high resolution.

### 3. Small-image upscaling

- In the budget path only: images below the minimum pixel floor (~512^2) are upscaled, capped at 2x linear so tiny images do not burn tokens pointlessly (larger regions should be re-cropped instead).

### 4. Built-in decision skill

- New builtin skill `image-workflow` (SKILL.md): tool-selection decision table, budget guidance (small preview / normal default / large detail; large on crops), coordinate discipline (normalized 0-1000, model coordinates are estimates, use reported width/height/was_resized to convert), and cost tips (image tokens dominate; crop before repeated full-image calls).

## Non-goals (follow-ups)

- 32px patch-grid snapping for Qwen-style providers
- Text label rendering in `draw_bbox` (needs font rasterization)
- Budget support for the `ImageAnalyzer` batch pre-analysis path

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

Start by tracing the existing analyze_image, view_image, optimize_image_with_size_limit, and ImageAnalyzer paths, including ImageLimits and coordinate_note handling. Then review how built-in SKILL.md files are registered. Done means budget presets, crop_image and draw_bbox, budget-path upscaling, and the image-workflow skill work together while legacy behavior remains unchanged.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
rust
Domaine
ai, computer-vision, tooling
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
Calme
Clarté
Plutôt claire
Accessibilité débutants
38/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.