ivrit-ai / ivrit-ai/transcriptor

Integrate handwriting collection (collect-handwriting.ivrit.ai) as a native tab in Transcriptor

Open
#23 1 comment 0 reactions 0 assignees View on GitHub
nice to have
Dominant language
TypeScript
Stars
1
Forks
3
Avg merge
1m
Merged PRs (30d)
1

Description

## Summary

Integrate the standalone handwriting collection app (currently deployed at `collect-handwriting.ivrit.ai`, repo [`ivrit-ai/ivrit-handwriting-collection`](https://github.com/ivrit-ai/ivrit-handwriting-collection)) into `transcriptor` as a new tab, so contributors can upload handwriting samples (with metadata) from the same site and account they use to transcribe.

This is a tracking issue for the integration. Decisions below reflect answers already given by the requester; a few secondary questions remain open (see bottom) and should be resolved before/during implementation.

## Current state (for context)

**`ivrit-handwriting-collection`** (standalone):
- FastAPI + server-rendered Jinja2 templates (multi-page flow: `/` → `/upload` → form → `/submit` → `/confirmation/{uuid}`), not a SPA.
- Its own Google OAuth (manual authorization-code flow in `routes/auth.py`), its own `SessionMiddleware`, its own CSRF token handling.
- No database — storage is JSON-file based: each submission is written to `uploads/{user_uuid}/{submission_uuid}/desc.json` plus the uploaded files, where `user_uuid` is derived deterministically from the user's email (`utils/user_uuid.email_to_user_uuid`).
- Captures form metadata (decade written, writer age range/gender, native language, legibility score, consent, notes) + browser metadata (screen size, timezone, user agent) alongside each upload.
- Maintains a simple stats cache (`total_contributors`, `total_pages`) via `utils/json_storage.py`.
- Local dev bypass: `FLASK_ENV=development` + `TS_USER_EMAIL=...`.

**`transcriptor`** (destination):
- FastAPI backend (`app/`) + React/TypeScript SPA (`frontend/`), PostgreSQL via SQLAlchemy/Alembic.
- Auth: production uses an `__Host-xhost_id` JWT cookie (issued by `auth.xhostd.com`, verified via JWKS in `app/api/deps.py`), which resolves to a `User` row (`google_sub`, `email`, `display_name`, `role`) via `get_or_create_user`. Dev mode auto-creates a fixed dev user — no separate OAuth flow lives in this app; auth is delegated to xhostd.
- Already has a consent system (`app/services/consent.py`, `ConsentGuard` on the frontend) gating `/work` behind a `contribution_license` consent type, versioned via `CONSENT_VERSION`.
- Frontend already has a multi-tab structure (`TopNav`, screens for Work/Leaderboard/Progress/Admin, with `AdminScreen` further split into `UsersTab`/`ReportsTab`/curate screens) — a new "Handwriting" tab fits this existing pattern.
- Images/files are served via `app/storage.py`; deploy (`scripts/deploy.py`) force-pushes a curated set of tracked paths (`app/`, `alembic/`, `scripts/`, etc.) to the xhostd git remote — anything new (routes, models, migrations) needs to live under those paths to be picked up automatically.

## Proposed technical approach

1. **Backend**: Add a new route module (e.g. `app/api/routes/handwriting.py`) mounted under `/api`, reusing `get_current_user` / `require_contribution_consent` from `app/api/deps.py` instead of the standalone app's own auth. Endpoints roughly mirror the old app's flow: upload files → submit metadata → confirmation, but as JSON APIs instead of server-rendered HTML.
2. **Data model**: Introduce SQLAlchemy models (e.g. `HandwritingSubmission`, `HandwritingFile`) + an Alembic migration, replacing the old JSON-file-per-submission approach, OR keep a JSON-sidecar-on-disk approach if the team prefers minimal modeling — **needs a decision** (see open questions).
3. **File storage**: Reuse `app/storage.py` conventions for where uploaded files land; port over `utils/file_handler.py` (file validation), `utils/browser_metadata.py`, and `utils/validators.py` logic from the old app largely as-is.
4. **Frontend**: New screen(s) under `frontend/src/screens/` (e.g. `HandwritingUploadScreen.tsx`) plus a nav entry in `TopNav`/`App.tsx`, following existing patterns (`AuthGuard`, `ConsentGuard`). Re-implement the old app's multi-step form (file picker → metadata form → confirmation) as a single React flow calling the new JSON endpoints.
5. **Consent**: Decide whether handwriting upload consent reuses the existing `contribution_license` consent type or needs a distinct one, since the old app's consent checkbox text is specific to handwriting collection.
6. **Stats**: Fold `total_contributors`/`total_pages` handwriting stats into the existing leaderboard/progress views, or keep them as a separate stat block on the new tab.
7. **Retire standalone app**: Once the tab is live, point `collect-handwriting.ivrit.ai` DNS/hosting at a redirect to the new tab URL on the transcriptor site, and archive/sunset the `ivrit-handwriting-collection` deploy.
8. **Data migration**: Existing submissions already stored under `uploads/{user_uuid}/{submission_uuid}/` in the standalone app need a migration plan into the new storage (see open questions).

## Suggested scope breakdown

- [ ] Design new DB schema / storage approach for handwriting submissions in `transcriptor`
- [ ] Alembic migration for new tables (if DB-backed)
- [ ] Backend API endpoints: upload, submit metadata, list/confirm own submissions
- [ ] Port file validation, browser-metadata capture, and form validators from `ivrit-handwriting-collection`
- [ ] Wire consent requirement (new or reused consent type)
- [ ] Frontend: new "Handwriting" tab/screen(s), nav entry, guards
- [ ] Migrate existing submission data (files + metadata) from the standalone app's `uploads/` into the new store
- [ ] Update `transcriptor` deploy script/docs if new paths are introduced
- [ ] Set up redirect from `collect-handwriting.ivrit.ai` to the new tab
- [ ] Sunset the standalone deploy of `ivrit-handwriting-collection`
- [ ] Update docs/READMEs in both repos to reflect the new architecture

## Open questions (not yet resolved — please advise)

1. **Storage model**: Model handwriting submissions as proper Postgres tables (consistent with the rest of `transcriptor`), or keep the disk + JSON-sidecar approach from the standalone app for simplicity? This affects effort significantly.
2. **Historical data migration**: Do existing submissions already collected by `collect-handwriting.ivrit.ai` need to be imported into the unified system, or can they stay archived separately (with only new uploads going through transcriptor)?
3. **Consent**: New `consent_type` specific to handwriting collection, or reuse the existing `contribution_license` consent that already gates `/work`?
4. **Stats/leaderboard**: Should handwriting contributions count toward the existing leaderboard/progress stats, or remain a separate metric shown only on the new tab?
5. **Redirect timing**: Should the redirect go live immediately when the tab ships, or after a transition/announcement period where both are reachable?
6. **Permissions**: Is handwriting upload open to any authenticated+consented user (same as transcription work), or does it need its own gating (e.g. admin-invited contributors only)?

## References

- Standalone app: https://collect-handwriting.ivrit.ai/
- Source: `ivrit-ai/ivrit-handwriting-collection`
- Destination: `ivrit-ai/transcriptor`

Contributor guide

No contributing guide indexed for this repository

Research direction

Resolve the open storage, migration, consent, stats, redirect, and permissions questions first. Then read app/api/deps.py, app/storage.py, the existing frontend navigation and guard patterns, and the standalone app's routes/auth.py plus validation utilities. Done means the integrated tab, APIs, data handling, consent, migration or archive plan, deployment transition, and documentation are complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, postgresql, python, react, sqlalchemy, typescript
Domain
api, authentication, backend, databases, devops, frontend, full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.