High-level background tasks
- Dominant language
- No language data
- Stars
- 39
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
The new background job system proposed in #66 is more-or-less in place (ckan/ckan#3165). It provides a low level interface for executing asynchronous jobs in separate worker processes. What it doesn't provide (by design) are high-level features that are often useful when dealing with background activities:
- Scheduling (e.g. run the same job each day at midnight or once at a specific time in the future)
- Information about past jobs, including failed jobs (e.g. exception messages, job durations)
I'd like to propose a high-level _task_ interface on top of the new _job_ system that provides this kind of functionality. Tasks would be persistent entities that are stored in PostgreSQL and could be manipulated similar to other CKAN objects like for example datasets. One would probably need to distinguish between a task (which may have a certain repetition frequency) and its executions (each of which can, for example, fail with a different error message).
Scheduling for RQ can be done via [rq-scheduler](https://github.com/ui/rq-scheduler).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the background job system proposed in #66 and the implementation referenced as ckan/ckan#3165. Compare its low-level worker interface with rq-scheduler, then define how persistent PostgreSQL tasks, schedules, executions, failures, and durations would be represented. Done means the high-level task design and scope are agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100