Move BATCH_DIR + NEON_MANIFESTS_DIR off node-local disk (spun off from #220 row 30)
- Dominant language
- Python
- Stars
- 0
- Forks
- 2
- Avg merge
- 10h 52m
- Merged PRs (30d)
- 42
Description
Spun off from #220 (Alpha 3 -- Scaffolding removal), row 30 -- "Move BATCH_DIR + NEON_MANIFESTS_DIR off node-local disk (Postgres-backed like job_sessions)". Both halves of this row turned out to need a real design decision rather than a mechanical swap, so tracking them here instead of rushing an approach inside the #220 batch.
## BATCH_DIR (text-service)
`text-service/main.py`'s `BATCH_DIR` (batch-download zips, `/batch-download/{id}`) lives on **text-service's own local disk** -- and unlike `NEON_MANIFESTS_DIR` below, text-service has **zero DB/Celery wiring today** (confirmed: no psycopg2 import anywhere in that service). Two real options, needing a decision before implementation:
1. **Give text-service its own DB access** -- add psycopg2 + `DATABASE_URL` wiring, store zip bytes in a new Postgres table mirroring `job_sessions`'s pattern. Self-contained, no new cross-service HTTP call, but a real new architectural coupling for a service that's deliberately been DB-free so far.
2. **Route zip bytes through the backend** -- text-service `POST`s the finished zip to a new backend endpoint after building it; backend stores it in `job_sessions`-style storage. Keeps text-service DB-free, but adds a new synchronous cross-service call and a new backend endpoint to design/secure (auth, size limits).
## NEON_MANIFESTS_DIR (backend)
Less obviously simple than it first looked: `mei_api.py` writes `{session_id}.jsonld` to `NEON_MANIFESTS_DIR`, but **nothing in this repo's Python code ever reads it back via an API route**. The embedded Neon editor bundle (`neon/deployment/scripts/editor.ts`) fetches it directly as a **static file** at a fixed relative URL -- `fetch('./samples/manifests/${name}.jsonld')`, resolved through `main.py`'s `/neon` `StaticFiles` mount (`landing-page/public/neon/`).
Moving this into Postgres therefore isn't just swapping the write path -- it means **replacing the StaticFiles-served path with an equivalent dynamic route** that:
- Serves at the exact same URL shape the Neon bundle already fetches (`/neon/samples/manifests/{id}.jsonld`), or
- Requires a submodule-side change to `editor.ts`'s fetch call (touches the `neon/` submodule, a separate repo)
Either path needs to preserve the StaticFiles behavior it's replacing (correct `Content-Type`, 404 on missing/expired manifest, no path-traversal surface) rather than just being "a route that returns JSON".
## Suggested next step
Pick one design per section above (can be decided independently -- they don't depend on each other), then implement as its own PR(s), same "one PR per row" convention as the rest of #220's batch.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.