CentreForDigitalHumanities / CentreForDigitalHumanities/Textcavator

Prevent full data download overuse

Open
#882 1 comment 1 reaction 0 assignees View on GitHub
on hold visualisation
Dominant language
Python
Stars
12
Forks
3
Avg merge
3d 3m
Merged PRs (30d)
9

Description

When we finish #707, users will be able to request term frequencies on the complete corpus, which can take up significant computation time. At the moment, we are just deploying this for the 12 people & parliament users, but if we want to use this on the main i-analyzer server, it would be good to have some kind of protection against a single user overwhelming the server with expensive computations.

(Note that we already have job queueing through celery and multiple workers. This issue is not about the infrastructure of handling many requests, just about preventing overuse.)

A simple solution is to only unlock the feature to users we have given permission. Could also be based on the download limit of the user. The default download limit happens to also be the cap for the term frequency graph.

A more involved idea: requests for visualisations already need to be linked to users accounts (to let them know when their requested visualisation is done), so we could say that users can only have one open request at a time (or maybe 3 or something, this could also be set per user/role). Note that this may cause issues when the user's requested visualisation never finishes due to a bug (or if they requested something by mistake), so this kind of requires the option to see your requested visualisation jobs and cancel them.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.