AcevedoR / AcevedoR/rpg-maestro
Cloud-readiness: rework YouTube ingestion into a durable queue + worker
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 0
- Avg merge
- 16m
- Merged PRs (30d)
- 7
Description
## Problem
The YouTube ingestion pipeline keeps job state in process memory, coordinates by one service polling another service's memory, and runs long CPU-bound work inside the HTTP-serving process. All three break with more than one instance or on an autoscaling runtime.
**Job state is per-process**
- `apps/audio-file-uploader/src/app/fileUpload/fromYoutube/upload-from-youtube-jobs-store.ts:6` — `readonly db = new Map<...>()`
- `apps/rpg-maestro/src/app/infrastructure/database.module.ts:17` — the maestro-side store is bound to `InMemoryTrackCreationFromYoutubeJobsStore`
Two instances hold disjoint job views; a client polling job status hits whichever instance the LB picks and may see nothing.
**The watcher is a per-instance polling loop**
- `apps/rpg-maestro/src/app/track-creation-from-youtube-jobs-watcher/track-creation-from-youtube-jobs-watcher.service.ts` — `wakeUp()` starts an in-process 5s loop (`SYNC_LOOP_INTERVAL_IN_MS = 5000`) that calls the uploader's `GET /upload/audio/from-youtube` and creates tracks
With N maestro instances there are N watchers racing on the same jobs: duplicate track creation, no idempotency key, no leader election. It also polls the uploader across the load balancer, so it only ever observes one uploader instance's in-memory jobs.
**Long work runs in the request-serving process**
- `apps/audio-file-uploader/src/app/app.controller.ts:36` — `uploadAudioFromYoutube` returns 202 immediately while an un-awaited `Promise.all` of ytdl download + ffmpeg transcode continues in the background
On Cloud Run / ECS / K8s-with-HPA that work is killed on scale-in or SIGTERM, silently. Neither `apps/rpg-maestro/src/app-bootstrap.ts` nor `apps/audio-file-uploader/src/main.ts` installs a graceful-shutdown hook.
## Proposed rework
- Durable job records in Firestore (or the DB) instead of a `Map`
- A real queue (Cloud Tasks / Pub-Sub) consumed by a **separate worker deployment** not autoscaled on HTTP traffic
- Idempotent handlers keyed on job id, with retries
- This removes `TrackCreationFromYoutubeJobsWatcher` entirely — the worker updates job state directly instead of one service polling another's memory
- Add `enableShutdownHooks()` / SIGTERM handling so in-flight work drains
Contributor guide
No contributing guide indexed for this repository
Research direction
Examine the in-memory job stores in apps/audio-file-uploader/src/app/fileUpload/fromYoutube/upload-from-youtube-jobs-store.ts and apps/rpg-maestro/src/app/infrastructure/database.module.ts. Review the polling watcher service at apps/rpg-maestro/src/app/track-creation-from-youtube-jobs-watcher/track-creation-from-youtube-jobs-watcher.service.ts and the HTTP handler in apps/audio-file-uploader/src/app/app.controller.ts. The goal is to replace these with a durable queue (Cloud Tasks/Pub-Sub) and a separate worker deployment, ensuring idempotent processing and graceful shutdown hooks. Done means the Map and polling loop are removed, jobs survive restarts, and no duplicate tracks are created.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, google-cloud, kubernetes, nodejs, typescript
- Domain
- backend, cloud, devops, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 20/100