vercel-labs / vercel-labs/json-render

Enable `catalog.prompt()`-generated system prompts support for smaller context window sizes

Open
#324 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
16.8k
Forks
901
Avg merge
3h 14m
Merged PRs (30d)
4

Description

The system prompt generated by catalog.prompt() is approximately 4,300 tokens for our catalog of 18 components.

This creates problems for small language models and models running locally on browsers, mobile devices, and other edge environments. Many of these models support relatively small context windows (at most ~4k tokens). A 4,300-token system prompt, alone, cannot fit inside this context window.

For now, we manually reduced the system prompt to approximately 2,000 tokens, so json-render can work on a SLM.

Desired outcome

Provide a way in PromptOptions to enable the caller of catalog.prompt() to generate a much smaller system prompt.

Implementation Ideas:

  • size option: a rough t-shirt size (sm, md, lg) for high level prompt length control, (where sm targets the prompt to ~1k tokens, md 2k tokens, and lg the current size)
  • sections options + size option: grandular control over the specificity at each section
  • template option (with sections + size hooks): allow user to add a handlebars-like template with preset helpers for sections. This is the most robust and flexible but could potentially introduce a new dependency.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with PromptOptions in packages/core/src/schema.ts and locate the catalog.prompt() implementation that assembles the system prompt. Compare the proposed size, sections, and template approaches against the existing prompt structure. Done means callers can request a substantially smaller prompt suitable for approximately 2k-token context budgets without losing the required catalog behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.