TemaDeveloper / TemaDeveloper/personal_planner

feat(onboarding): Attach images in onboarding chat for vision-based understanding

Open
#65 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement help wanted
Dominant language
TypeScript
Stars
2
Forks
4
PR merge metrics
No merged PRs in 30d

Description

Size: L — headline onboarding improvement.

Why

Onboarding is a pure text interview (ONBOARDING_SYSTEM_PROMPT, conversational-onboarding.tsx — the composer is a single text input, no attach affordance). People can describe their life far faster by showing it: a photo of their paper planner, a screenshot of a budgeting spreadsheet, a gym program, a class timetable, a meds list, a whiteboard of goals. Letting them attach images makes the derived life-profile richer and the generated sections fit better on the first try.

Scope

  1. Client (src/components/onboarding/conversational-onboarding.tsx): add an attach button + thumbnail previews + remove; allow paste/drag-drop; send image refs alongside the text turn.
  2. Upload: reuse the existing private-Blob path (/api/notes/upload: image-only, 10MB, ownership-scoped serving via /api/notes/blob) — or a parallel onboarding/<userId>/... namespace. Don't send raw base64 through the chat body.
  3. API (src/app/api/profile/chat/route.ts): accept images on the turn; pass them to the vision-capable extraction + reply calls (depends on multimodal callAI — see the multimodal-callAI issue).
  4. Extraction (src/lib/profile/facet-extract.ts): feed images so facets are derived from what's shown, not just typed.
  5. Update ONBOARDING_SYSTEM_PROMPT to invite "type or show me" and to reason over attached images.

Notes

  • Vision inference on multiple images is slow — respect/raise maxDuration (currently 60s) and stream or show progress.
  • Privacy: onboarding images can be sensitive (finances, health) — keep them private-access, scoped to the user, and consider deleting them after profile extraction.

Depends on: multimodal callAI. Pairs with: seed-from-artifact, confirm-understanding.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/components/onboarding/conversational-onboarding.tsx, /api/notes/upload, /api/notes/blob, src/app/api/profile/chat/route.ts, and src/lib/profile/facet-extract.ts. First verify the multimodal callAI dependency and existing private Blob flow. Done means image selection, previews, paste and drop work through extraction and replies while access remains user-scoped and the onboarding prompt handles images.

Written by the indexing model from the issue text.

Assessment

Tech stack
nextjs, react, typescript
Domain
ai, api, backend, frontend, full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.