[Enhancement]: Compress raster images at every ingestion point and add OCR for text-only models
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 19.9k
- Forks
- 2.3k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 30
Description
Summary
jcode stores images in session history and sends them to providers at their
original size. A phone screenshot taken via adb exec-out screencap -p is a
1.4–1.6 MB PNG (1080×2400), which becomes ~1.9–2.2 MB of base64 per image in
the transcript and on the provider wire on every turn. Sessions with a handful
of screenshots balloon to multiple MB and the prompt-cache prefix grows on
every image-bearing message. Additionally, text-only models (DeepSeek) receive
no image content at all: jcode already replaces ContentBlock::Image with
[Image omitted: this provider/model does not support image input] for those
providers, so the agent cannot see the screen at all.
Impact
Verified on a real QA run: a single read shots/home.png (1.4 MB)
added ~1.9 MB of base64 to the session journal and the DeepSeek request was
sent without the image (the model answered "The model can't process images"
and fell back to uiautomator dump). Any agent doing mobile/desktop QA with
screenshots hits both problems: session bloat + invisible screens for
text-only models.
Changes (implemented and verified locally)
1. Vision image budget in crates/jcode-base/src/provider/image_clamp.rs
Previously the clamp only enforced Anthropic's hard per-request caps (10 MB
base64 / 8000 px / 2000 px many-image). Added a small per-image budget:
- max edge 1568 px (the sweet spot every vision provider downsamples to internally)
- max base64 payload 256 KB
- JPEG quality 80 (4:2:2 chroma subsampling keeps UI text crisp)
- overridable per-daemon via env without rebuild:
JCODE_IMAGE_MAX_EDGE,JCODE_IMAGE_MAX_BYTES,JCODE_IMAGE_JPEG_QUALITY
clamp_outbound_images now uses the user budget clamped to the provider's
absolute safety caps, so a misconfigured env can never produce a request
Anthropic rejects.
We deliberately use JPEG rather than WebP: the image crate's built-in WebP
encoder is lossless-only (VP8L), which is larger than JPEG for photos and
busy screenshots, and lossy WebP would pull in a libwebp native dependency.
2. Compress at every ingestion point (not just on the wire)
New compress_image_b64(media_type, data) -> Option<(String, String)> skips
vector formats (image/svg+xml, image/svg, *+xml) so vector artwork
never gets rasterized. The following ingestion sites now compress raster
images as they enter the transcript:
readtool (crates/jcode-app-core/src/tool/read.rs) — compress at read timetool_output_to_content_blocks(agent/tools.rs) — every tool with attached
images (browser screenshots, screen captures, etc.)generated_image_payload(crates/jcode-base/src/message.rs) — provider-
generated imagesappend_user_context_message(agent/turn_execution.rs) — user-attached imagesinject_soft_interrupts(agent/interrupts.rs) — soft-interrupt images
The on-the-wire clamp stays as a safety net for any block that bypasses these
paths.
3. OCR for text-only models (read tool)
When tesseract is present (not in the default image), the read tool runs
OCR over the raw image bytes and appends the recognized text to the tool
result, so a text-only model sees the screen content as text. Env:
JCODE_IMAGE_OCR=0 to disable, JCODE_IMAGE_OCR_LANGS (default rus+eng).
Verification
- 15/15
image_clampunit tests pass, including new ones for
compress_image_b64(large raster compresses to JPEG; SVG/vector passes
through untouched). - Real run:
jcode runreading a 1.6 MB phone screenshot →
Image: /tmp/ocr_test.png (162.2 KB), agent returned the on-screen text via
OCR, session journal storedimage/jpeg(167 KB vs 2.3 MB for the old
uncompressed session). - On-the-wire: image payload drops from ~1.9 MB to ~200 KB of base64.
Environment
- jcode v0.68.0 (patched on top of
fcf5390), Linux x86_64. - Provider: DeepSeek (text-only) and vision providers benefit equally.
imagecrate features:png,jpeg(no new native deps).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the listed ingestion points, especially crates/jcode-base/src/provider/image_clamp.rs and crates/jcode-app-core/src/tool/read.rs, plus agent/tools.rs and agent/turn_execution.rs. Run the image_clamp unit tests first, then inspect the read-tool OCR and environment-variable paths. Done means all 15 tests pass and the stated compression, vector pass-through, OCR, and text-only-model behavior is verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 28/100