CommandCodeAI / CommandCodeAI/command-code

vision input fails

未關閉
#559 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

還沒有人認領這個 Issue。

主要語言
沒有語言資料
星號
4k
分支
350
PR 合併指標
30 天內沒有已合併 PR

描述

Summary

Vision input currently works only through interactive TUI clipboard-paste and drag-drop (detectDroppedImageFile / detectClipboardImage → Jimp → base64 → {type:"image"}).
Environment

  • cmd v0.38.2
  • Installed via npm i -g command-code (mise)
  • macOS (darwin)
  • Model: Claude Sonnet 4 (claude-4-sonnet-20250514) — confirmed inputModalities: ["text","image"]
    Reproduce

1. Put any JPEG or PNG in the working directory

2. Run:

cmd -p "Describe what you see in test.jpg"
Result: Model responds that it cannot visually inspect the image. Same result for PNG, GIF, WEBP.
Root cause (from inspecting dist/index.mjs)

  1. read_file returns text-only output for all binary files
    src/tools/read-file/read-file.ts → formatOutput:
    function formatOutput(e) {
    if ("binary" === e.contentType)
    return [..., Binary file: ${e.fileType}].join("\n");
    // Model only ever sees: "Binary file: image/jpeg"
    ...
    }
    No {type: "image"} content block is ever created from a file read. The raw bytes are read into a Buffer but discarded.
  2. @filepath references silently skip binary files
    src/utils/file-references.ts → processFileReferences:
    if ("binary" === i.contentType) continue; // images vanish entirely
  3. AVIF and other modern formats fall through to the text path
    getFileType uses a hardcoded extension map (Ib) with no .avif entry:
    Ib = { ".png":"image/png", ".jpg":"image/jpeg", ".jpeg":"image/jpeg",
    ".gif":"image/gif", ".webp":"image/webp", ".svg":"image/svg+xml",
    ".bmp":"image/bmp", ".ico":"image/x-icon", ... }
    // No .avif, .heic, .heif, .tiff
    Since getFileType(".avif") returns "unknown", and isBinaryFile("unknown") returns false (it only matches known MIME values), AVIF files are read as UTF-8 text — producing garbled output rather than being flagged as binary.
  4. Vision content blocks only exist in the TUI paste/drag-drop path
    src/utils/clipboard-image.ts → detectDroppedImageFile → Jimp resize → base64 → {type:"image"}. This path is a React/Ink interactive handler and is not reachable in headless mode.
Expected Behavior

..

Actual Behavior

..

Steps to reproduce the issue

..

Command Code Version

latest

Operating System

macOS

Terminal/IDE

No response

Shell

No response

Additional context

No response

貢獻指南

這個儲存庫沒有索引到貢獻指南

從這裡開始

  1. 先讀完整個 Issue,再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫,在一個分支上完成修改。
  4. 送出 Pull Request,並在描述裡引用這個 Issue 編號。

研究方向

從 src/tools/read-file/read-file.ts 和 src/utils/file-references.ts 開始,接著將它們的二進位處理與 src/utils/clipboard-image.ts 進行比較。針對本機影像使用 cmd -p 重現,並追蹤檔案如何到達模型;完成標準是,支援的影像檔案(包括 issue 中提到的格式)在 headless mode 下產生 vision content,而不是只有文字或被跳過。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
typescript
領域
ai, cli
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
冷清
描述清晰度
基本清楚
新手友好度
48/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。