denoland / denoland/deploy_feedback

Best way to use a Database-backed queue such as Graphile Worker?

Open
#789 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
No language data
Stars
79
Forks
5
PR merge metrics
No merged PRs in 30d

Description

### What problem are you trying to solve?

I want to run a 'worker' process backed by Postgres Database. The use-case is very similar to Deno-Queue, but using Postgres instead.

Database-backed worker queues are a pretty common and useful pattern for servers. Currently I am using [Graphile Worker](https://worker.graphile.org ) as an executor (consumes from the queue and processes the data), but there are other libraries, the question would be useful for people using other libraries or other databases.

The problem that I see is that Deno deployments are 'turned-off' if there are no incoming HTTP requests, and a worker-queue does not rely on HTTP requests, instead it uses other methods such as listening for NOTIFY from the database along with polling.

So how could an isolate await an async function for a long time even when there are no incoming HTTP requests?

It is worth noting that such a Database queue is useful because services in other places, and other programming languages, are able to append messages. The queues also include a lot of useful features, such as exponential backoff, tracking state, deadletter queue, etc. These functionalities are a lot more than offered by simpler queues, such as DenoKV.

### Describe the solution you'd like

Some way to spin up an isolate upon a Postgres NOTIFY ([PGNotify](https://www.postgresql.org/docs/current/libpq-notify.html)), the same way that an HTTP request spins up an isolate.

### Describe alternatives you've considered

- Deno Cron to poll for new jobs. But this is very inefficient and introduces latency.
- Sending an empty http request to the deployment so that it 'wakes up' and checks the queue for new messages.

Would these work-arounds actually work for this problem?

Any suggestion/alternative to achieve a worker queue would be very valuable to know of.

### Documentation, Adoption, Migration Strategy

_No response_

References:
- https://github.com/denoland/deploy_feedback/issues/143
- https://deno.com/blog/anatomy-isolate-cloud

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.