lookit / lookit/lookit-api

video download all: prevent simultaneous jobs for same study

Open
#1,847 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
12
Forks
21
Avg merge
5d 19h
Merged PRs (30d)
5

Description

TL; DR

The video download all task can be very resource intensive, and we allow two jobs to run in parallel. We should prevent a video-download-all task from running while this task is still in progress for the same study.

Description

The video download all task can be very time/resource intensive, and researchers might click the 'download all' button again before the first job is finished. Also, although we have recently improved the video-download-all task to make it use less memory, our system is still sensitive to the CPU load from zipping a large amount of video data. We recently had a site outage due to this task running twice (in parallel) for the same study, which had a large amount of video data. We can reduce the chances of two simultaneous large jobs by preventing the task from starting again while another job is still in progress for that same study.

Implementation

On the responses -> videos page, we should be able to check celery's active/queued jobs to see if there's a video download all task for that particular study, and if so, disable the 'download all' button (perhaps with a message saying that this job is already in progress/queued).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the responses → videos page and the video-download-all task, then inspect how Celery exposes active or queued jobs for a study. Done means a second download-all request for the same study is prevented while the first is active or queued, and the button shows an in-progress message.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, frontend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.