ckan / ckan/ideas

Activity Stream connected to ELK/Splunk for analysis

Open
#209 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
39
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Admins should be able to analyse key activity in CKAN. This means creation / editing / deleting a dataset or organization, or user permissions being granted.

Example use cases:
* generate stats on numbers of datasets being created / edited per week, broken down by user, organization, number of resources, resource hostname (lots of scope for this)
* an admin behaves badly, so the system admin wants a full audit trail of their actions, including the full dataset changes.
* report on which organization's datasets are being updated and which are not

Whilst the CKAN logs are mainly unstructured data, the Activity Stream is good, but incomplete and not easily accessible in the database.

Whilst we can continue to show the basic Activity Stream in CKAN's web interface, let's take advantage of external data analysis software to allow more advanced exploration, searching, filtering and graphing, rather than trying to build it into CKAN.

I propose:
* I complete the Activity Stream functionality so that every create/edit/delete of package/group/organization/user/member is logged in the Activity Stream. We can also record if an action is done on the web interface or via the API.
* Activity Stream can be linked into data analysis software to be sliced and diced. The simplest way would be to add an option to dump the Activity Stream data to a JSON log file, which can be easily loaded into lots of analysis software. For example this could be shipped in real time to an [ELK stack](https://www.elastic.co/products) and explored and graphed by pointing and clicking. Or enterprises can do similar things with Splunk or Sumo Logic.

(An alternative to a JSON log file would be getting the analysis software to talk to Postgres directly using JDBC, and setting up queries to do lots of joins to get the full Activity Stream. This relies a lot more on the analysis software having this capability and setting up the queries in it. I think it would be better to do the join query work in CKAN, with the result in JSON log, which is really much more flexible, and easily shipped.)

Comments v. welcome!

I'm working with OpenGov to explore this, so in particular please chip in @jqnatividad @jhinds

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing CKAN's existing Activity Stream and the web-interface and API paths that record actions. Compare the proposed event coverage and JSON-log export with the existing stream; done would require an agreed scope, complete audit events, and a practical path to ELK or similar analysis tools.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, json, postgresql
Domain
analytics, backend, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.