TemaDeveloper / TemaDeveloper/personal_planner
feat(ai): Multimodal callAI — image content blocks for Claude/Gemini/OpenAI/Mistral
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 2
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
Size: M — enabling infra (prerequisite for onboarding image attachments #65).
Problem
callAI(provider, apiKey, systemPrompt, userMessage) (src/lib/ai.ts:278-333) is text-only — it passes a plain string as the sole user message for all four providers. All the models actually configured are vision-capable (Claude claude-sonnet-4, Gemini gemini-2.0-flash, Mistral mistral-large-latest, OpenAI gpt-4o-mini), so nothing but the wrapper prevents multimodal input. Facet extraction (facet-extract.ts:84) and the onboarding reply (profile/chat/route.ts:67) both call this text-only path.
Scope
Add a multimodal variant (e.g. callAIVision(provider, apiKey, system, text, images: {mime, dataOrUrl}[])) that builds provider-specific image content blocks:
- Claude:
content: [{type:"text"}, {type:"image", source:{type:"base64"|"url", ...}}] - OpenAI:
content: [{type:"text"}, {type:"image_url", image_url:{url}}] - Gemini:
contentsparts withinlineData/fileData - Mistral:
image_urlcontent parts (pixtral-capable model — may need a model bump)
Keep callAI as the text-only fast path; the vision call is opt-in. Cap image count/size and downscale server-side to control token cost. Handle providers/models that can't do vision by falling back to text + a note.
Blocks: the onboarding attachment feature.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/lib/ai.ts:278-333 and inspect the existing provider-specific callAI handling, then trace callers at facet-extract.ts:84 and profile/chat/route.ts:67. Done means an opt-in multimodal path supports the four listed providers, preserves the text-only path, limits image size/count, and falls back to text with a note when vision is unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nextjs, typescript
- Domain
- ai, api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100