Feature/improvement suggestions for ckanext-realtime
- Dominant language
- No language data
- Stars
- 39
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
I have been working on ckanext-realtime (https://github.com/alexandrainst/ckanext-realtime) for some time now and I have reached the initial goal for it. If you enable this extension on your CKAN instance, client applications can monitor changes to datastores in realtime. In the picture below you can see how the realtime datastores are enabled with this plugin.

1. In order to emit realtime events, each "realtime datastore" table has a PostgreSQL trigger which sends NOTIFY events to datastore_listener script. The insertion of the trigger to datastore tables happens automatically: first time someone is interested to monitor a particular datastore, that datastores gets the trigger.
2. datastore_listener script sends datastore events (insert/update/delete) to ckan through Action API function _realtime_broadcast_event_.
3. CKAN and WebSocket server(s) are integrated using Redis Pub/Sub messaging where CKAN is the publisher and WebSocket server(s) are the subscribers. This way, every event sent to _realtime_broadcast_event_ API function ends up in WebSocket servers.
4. Clients can subscribe to realtime events of certain datastores (pub/sub) using CkanRT.js library (included in ckanext-realtime repo).
So, right now it is possible for client apps to observe insert/delete/update events of certain datastores. You can find a demo app here http://gatesense.com/realtime/examples/ex2/ .
Other things that could be implemented:
- dataset level realtime events (new resources in datasets, dataset metadata change, etc.)
- direct integration into CKAN UI
- access control
- anything else?
The project was initiated by Alexandra Institute (http://www.alexandra.dk/uk/pages/default.aspx) and later adopted by Gatesense community (http://gatesense.com/) where Alexandra is one of the contributors. It is an effort to research realtime data publishing in the context of IoT and eventually gear CKAN towards being "IoT enabled" (though this may require additional extensions). That said, this extension has not been used on "real world applications" yet and it may not cover your specific use case. Therefore, we welcome community's collaborations and suggestions on how to improve it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the ckanext-realtime architecture described in the issue, especially the datastore_listener script, the _realtime_broadcast_event_ Action API function, and CkanRT.js. Choose one proposed area—dataset events, CKAN UI integration, or access control—then define its scope and acceptance criteria; the issue does not currently specify what completion should look like.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, postgresql, redis
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100