DDMAL / DDMAL/mothra

Move BATCH_DIR + NEON_MANIFESTS_DIR off node-local disk (spun off from #220 row 30)

Open
#230 0 comments 0 reactions 0 assignees View on GitHub
decision-needed landing mothra-text
Dominant language
Python
Stars
0
Forks
2
Avg merge
10h 52m
Merged PRs (30d)
42

Description

Spun off from #220 (Alpha 3 -- Scaffolding removal), row 30 -- "Move BATCH_DIR + NEON_MANIFESTS_DIR off node-local disk (Postgres-backed like job_sessions)". Both halves of this row turned out to need a real design decision rather than a mechanical swap, so tracking them here instead of rushing an approach inside the #220 batch.

## BATCH_DIR (text-service)

`text-service/main.py`'s `BATCH_DIR` (batch-download zips, `/batch-download/{id}`) lives on **text-service's own local disk** -- and unlike `NEON_MANIFESTS_DIR` below, text-service has **zero DB/Celery wiring today** (confirmed: no psycopg2 import anywhere in that service). Two real options, needing a decision before implementation:

1. **Give text-service its own DB access** -- add psycopg2 + `DATABASE_URL` wiring, store zip bytes in a new Postgres table mirroring `job_sessions`'s pattern. Self-contained, no new cross-service HTTP call, but a real new architectural coupling for a service that's deliberately been DB-free so far.
2. **Route zip bytes through the backend** -- text-service `POST`s the finished zip to a new backend endpoint after building it; backend stores it in `job_sessions`-style storage. Keeps text-service DB-free, but adds a new synchronous cross-service call and a new backend endpoint to design/secure (auth, size limits).

## NEON_MANIFESTS_DIR (backend)

Less obviously simple than it first looked: `mei_api.py` writes `{session_id}.jsonld` to `NEON_MANIFESTS_DIR`, but **nothing in this repo's Python code ever reads it back via an API route**. The embedded Neon editor bundle (`neon/deployment/scripts/editor.ts`) fetches it directly as a **static file** at a fixed relative URL -- `fetch('./samples/manifests/${name}.jsonld')`, resolved through `main.py`'s `/neon` `StaticFiles` mount (`landing-page/public/neon/`).

Moving this into Postgres therefore isn't just swapping the write path -- it means **replacing the StaticFiles-served path with an equivalent dynamic route** that:
- Serves at the exact same URL shape the Neon bundle already fetches (`/neon/samples/manifests/{id}.jsonld`), or
- Requires a submodule-side change to `editor.ts`'s fetch call (touches the `neon/` submodule, a separate repo)

Either path needs to preserve the StaticFiles behavior it's replacing (correct `Content-Type`, 404 on missing/expired manifest, no path-traversal surface) rather than just being "a route that returns JSON".

## Suggested next step

Pick one design per section above (can be decided independently -- they don't depend on each other), then implement as its own PR(s), same "one PR per row" convention as the rest of #220's batch.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.