ckan / ckan/ideas

High-level background tasks

Open
#180 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
39
Forks
1
PR merge metrics
No merged PRs in 30d

Description

The new background job system proposed in #66 is more-or-less in place (ckan/ckan#3165). It provides a low level interface for executing asynchronous jobs in separate worker processes. What it doesn't provide (by design) are high-level features that are often useful when dealing with background activities:
- Scheduling (e.g. run the same job each day at midnight or once at a specific time in the future)
- Information about past jobs, including failed jobs (e.g. exception messages, job durations)

I'd like to propose a high-level _task_ interface on top of the new _job_ system that provides this kind of functionality. Tasks would be persistent entities that are stored in PostgreSQL and could be manipulated similar to other CKAN objects like for example datasets. One would probably need to distinguish between a task (which may have a certain repetition frequency) and its executions (each of which can, for example, fail with a different error message).

Scheduling for RQ can be done via [rq-scheduler](https://github.com/ui/rq-scheduler).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the background job system proposed in #66 and the implementation referenced as ckan/ckan#3165. Compare its low-level worker interface with rq-scheduler, then define how persistent PostgreSQL tasks, schedules, executions, failures, and durations would be represented. Done means the high-level task design and scope are agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.