CommandCodeAI / CommandCodeAI/command-code

vision input fails

Open
#559 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
No language data
Stars
4k
Forks
350
PR merge metrics
No merged PRs in 30d

Description

Summary

Vision input currently works only through interactive TUI clipboard-paste and drag-drop (detectDroppedImageFile / detectClipboardImage → Jimp → base64 → {type:"image"}).
Environment

  • cmd v0.38.2
  • Installed via npm i -g command-code (mise)
  • macOS (darwin)
  • Model: Claude Sonnet 4 (claude-4-sonnet-20250514) — confirmed inputModalities: ["text","image"]
    Reproduce

1. Put any JPEG or PNG in the working directory

2. Run:

cmd -p "Describe what you see in test.jpg"
Result: Model responds that it cannot visually inspect the image. Same result for PNG, GIF, WEBP.
Root cause (from inspecting dist/index.mjs)

  1. read_file returns text-only output for all binary files
    src/tools/read-file/read-file.ts → formatOutput:
    function formatOutput(e) {
    if ("binary" === e.contentType)
    return [..., Binary file: ${e.fileType}].join("\n");
    // Model only ever sees: "Binary file: image/jpeg"
    ...
    }
    No {type: "image"} content block is ever created from a file read. The raw bytes are read into a Buffer but discarded.
  2. @filepath references silently skip binary files
    src/utils/file-references.ts → processFileReferences:
    if ("binary" === i.contentType) continue; // images vanish entirely
  3. AVIF and other modern formats fall through to the text path
    getFileType uses a hardcoded extension map (Ib) with no .avif entry:
    Ib = { ".png":"image/png", ".jpg":"image/jpeg", ".jpeg":"image/jpeg",
    ".gif":"image/gif", ".webp":"image/webp", ".svg":"image/svg+xml",
    ".bmp":"image/bmp", ".ico":"image/x-icon", ... }
    // No .avif, .heic, .heif, .tiff
    Since getFileType(".avif") returns "unknown", and isBinaryFile("unknown") returns false (it only matches known MIME values), AVIF files are read as UTF-8 text — producing garbled output rather than being flagged as binary.
  4. Vision content blocks only exist in the TUI paste/drag-drop path
    src/utils/clipboard-image.ts → detectDroppedImageFile → Jimp resize → base64 → {type:"image"}. This path is a React/Ink interactive handler and is not reachable in headless mode.
Expected Behavior

..

Actual Behavior

..

Steps to reproduce the issue

..

Command Code Version

latest

Operating System

macOS

Terminal/IDE

No response

Shell

No response

Additional context

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/tools/read-file/read-file.ts and src/utils/file-references.ts, then compare their binary handling with src/utils/clipboard-image.ts. Reproduce with cmd -p against a local image and trace how the file reaches the model; done means supported image files, including the formats named in the issue, produce vision content in headless mode instead of text-only or being skipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.