denoland / denoland/deploy_feedback
Best way to use a Database-backed queue such as Graphile Worker?
- Dominant language
- No language data
- Stars
- 79
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
### What problem are you trying to solve?
I want to run a 'worker' process backed by Postgres Database. The use-case is very similar to Deno-Queue, but using Postgres instead.
Database-backed worker queues are a pretty common and useful pattern for servers. Currently I am using [Graphile Worker](https://worker.graphile.org ) as an executor (consumes from the queue and processes the data), but there are other libraries, the question would be useful for people using other libraries or other databases.
The problem that I see is that Deno deployments are 'turned-off' if there are no incoming HTTP requests, and a worker-queue does not rely on HTTP requests, instead it uses other methods such as listening for NOTIFY from the database along with polling.
So how could an isolate await an async function for a long time even when there are no incoming HTTP requests?
It is worth noting that such a Database queue is useful because services in other places, and other programming languages, are able to append messages. The queues also include a lot of useful features, such as exponential backoff, tracking state, deadletter queue, etc. These functionalities are a lot more than offered by simpler queues, such as DenoKV.
### Describe the solution you'd like
Some way to spin up an isolate upon a Postgres NOTIFY ([PGNotify](https://www.postgresql.org/docs/current/libpq-notify.html)), the same way that an HTTP request spins up an isolate.
### Describe alternatives you've considered
- Deno Cron to poll for new jobs. But this is very inefficient and introduces latency.
- Sending an empty http request to the deployment so that it 'wakes up' and checks the queue for new messages.
Would these work-arounds actually work for this problem?
Any suggestion/alternative to achieve a worker queue would be very valuable to know of.
### Documentation, Adoption, Migration Strategy
_No response_
References:
- https://github.com/denoland/deploy_feedback/issues/143
- https://deno.com/blog/anatomy-isolate-cloud
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.