AcevedoR / AcevedoR/rpg-maestro
Cloud-readiness: rework YouTube ingestion into a durable queue + worker
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 0
- Avg merge
- 16m
- Merged PRs (30d)
- 7
Description
## Problem
The YouTube ingestion pipeline keeps job state in process memory, coordinates by one service polling another service's memory, and runs long CPU-bound work inside the HTTP-serving process. All three break with more than one instance or on an autoscaling runtime.
**Job state is per-process**
- `apps/audio-file-uploader/src/app/fileUpload/fromYoutube/upload-from-youtube-jobs-store.ts:6` — `readonly db = new Map<...>()`
- `apps/rpg-maestro/src/app/infrastructure/database.module.ts:17` — the maestro-side store is bound to `InMemoryTrackCreationFromYoutubeJobsStore`
Two instances hold disjoint job views; a client polling job status hits whichever instance the LB picks and may see nothing.
**The watcher is a per-instance polling loop**
- `apps/rpg-maestro/src/app/track-creation-from-youtube-jobs-watcher/track-creation-from-youtube-jobs-watcher.service.ts` — `wakeUp()` starts an in-process 5s loop (`SYNC_LOOP_INTERVAL_IN_MS = 5000`) that calls the uploader's `GET /upload/audio/from-youtube` and creates tracks
With N maestro instances there are N watchers racing on the same jobs: duplicate track creation, no idempotency key, no leader election. It also polls the uploader across the load balancer, so it only ever observes one uploader instance's in-memory jobs.
**Long work runs in the request-serving process**
- `apps/audio-file-uploader/src/app/app.controller.ts:36` — `uploadAudioFromYoutube` returns 202 immediately while an un-awaited `Promise.all` of ytdl download + ffmpeg transcode continues in the background
On Cloud Run / ECS / K8s-with-HPA that work is killed on scale-in or SIGTERM, silently. Neither `apps/rpg-maestro/src/app-bootstrap.ts` nor `apps/audio-file-uploader/src/main.ts` installs a graceful-shutdown hook.
## Proposed rework
- Durable job records in Firestore (or the DB) instead of a `Map`
- A real queue (Cloud Tasks / Pub-Sub) consumed by a **separate worker deployment** not autoscaled on HTTP traffic
- Idempotent handlers keyed on job id, with retries
- This removes `TrackCreationFromYoutubeJobsWatcher` entirely — the worker updates job state directly instead of one service polling another's memory
- Add `enableShutdownHooks()` / SIGTERM handling so in-flight work drains
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.