GCWing / GCWing/OpenBitFun

[Optimize]: Resolution budgets, crop/bbox coordinate tools, and image-workflow guidance for vision tools

Đang mở
#2,248 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
enhancement
Ngôn ngữ chính
Rust
Star
2.2k
Fork
229
Merge trung bình
2 giờ 46 phút
Pull request đã merge (30 ngày)
577

Mô tả

## Summary

Upgrade BitFun's built-in vision tools with four capabilities borrowed from Qwen-MM-Plugins' design:

1. **Resolution budget presets** for `analyze_image` / `view_image` (`small` / `normal` / `large`)
2. **Coordinate closed loop**: new `crop_image` and `draw_bbox` tools using 0-1000 normalized coordinates
3. **Small-image upscaling** to a minimum pixel floor (capped at 2x linear) so tiny crops stay legible for VLMs
4. **Built-in decision skill** (`image-workflow`) teaching the agent when to view vs analyze vs crop, and how to pick a budget

## Background

The current vision path is two fixed-limit tools plus a batch pre-analysis path:

- `analyze_image` sends the image + prompt to the configured image-understanding model. Every call pays full image tokens, and the only size control is the hard provider cap (1MB tool cap / provider dimension limits). There is no way for the agent to trade detail against cost.
- `view_image` attaches an image to the primary model context, also with no resolution control.
- `optimize_image_with_size_limit` never upscales, so a tiny crop (e.g. a 64x64 region) is sent as-is and OCR/detail suffers.
- The only coordinate handling is a prose `coordinate_note` warning the model that positions in the analysis are estimates. There is no tool to actually crop or annotate a static image, so the estimate cannot be turned into an action loop.
- Tool descriptions are one-liners; the model has no guidance on which tool to use when.

Qwen-MM-Plugins solves these with token-budget-driven resolution presets (256/1024/2048 visual tokens), a `crop`/`draw_bbox` pair that consumes grounding output in the same 0-1000 normalized coordinate system, an upscale floor, and declarative SKILL.md decision documents.

## Proposed changes

### 1. Budget presets (optional parameter, legacy behavior unchanged)

- Add `ImageBudget` (small/normal/large) mapping to pixel targets via token budgets (256/1024/2048 x 32^2 = ~512^2 / ~1024^2 / ~1448^2), clamped by the provider `ImageLimits` ceiling.
- Add an optional `budget` parameter to `analyze_image` and `view_image`. When omitted, the existing path runs unchanged (no behavior change for existing callers/configs).
- Fix the `resize_note` wording which currently hard-codes "downscaled" and would misreport upscaling.

### 2. Coordinate closed loop: `crop_image` + `draw_bbox`

- `crop_image(path, box[x1,y1,x2,y2 in 0-1000], output_path?)`: validates and clamps the box, saves the cropped region next to the source by default, and returns pixel-mapped coordinates. Attaches a preview when the primary model supports multimodal tool output; degrades to a text summary otherwise.
- `draw_bbox(path, boxes[{bbox,label?}], output_path?)`: draws rectangles with an adaptive line width and a fixed palette. v1 draws boxes only (no text labels, no new dependencies).
- Both support remote workspace paths (read + write through workspace filesystem services).
- Closed loop: the model reports a region in normalized coordinates, `crop_image` cuts it, then `analyze_image(crop_path, budget="large")` re-inspects the region at high resolution.

### 3. Small-image upscaling

- In the budget path only: images below the minimum pixel floor (~512^2) are upscaled, capped at 2x linear so tiny images do not burn tokens pointlessly (larger regions should be re-cropped instead).

### 4. Built-in decision skill

- New builtin skill `image-workflow` (SKILL.md): tool-selection decision table, budget guidance (small preview / normal default / large detail; large on crops), coordinate discipline (normalized 0-1000, model coordinates are estimates, use reported width/height/was_resized to convert), and cost tips (image tokens dominate; crop before repeated full-image calls).

## Non-goals (follow-ups)

- 32px patch-grid snapping for Qwen-style providers
- Text label rendering in `draw_bbox` (needs font rasterization)
- Budget support for the `ImageAnalyzer` batch pre-analysis path

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start by tracing the existing analyze_image, view_image, optimize_image_with_size_limit, and ImageAnalyzer paths, including ImageLimits and coordinate_note handling. Then review how built-in SKILL.md files are registered. Done means budget presets, crop_image and draw_bbox, budget-path upscaling, and the image-workflow skill work together while legacy behavior remains unchanged.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
rust
Lĩnh vực
ai, computer-vision, tooling
Loại issue
Tính năng
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
38/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.