lookit / lookit/lookit-api

Large video download all tasks can hold up task queue

Open
#1,745 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug Performance Tuning Researcher
Dominant language
Python
Stars
12
Forks
21
Avg merge
5d 19h
Merged PRs (30d)
5

Description

## Description

The 'download all videos' task runs on the builder pod and is part of the 'builder' queue, which also contains:
- EFP builds (ember_build_and_gcp_deploy)
- Data downloads (build_framedata_dict)

With large video files and/or numbers of files, a single 'download all videos' task can take a long time (e.g. 2 hours for my load test study on staging, which contains 10,000 short videos). There's no limit to how many of these tasks can be triggered simultaneously or in close succession. The 'builds' queue can run 2 tasks at a time, so a long-running task will hold up any other tasks in the same queue. This could become disruptive and confusing to researchers who sometimes have to wait a very long time for tasks like re-building an EFP study or downloading frame data.

## Possible solutions

- Move the 'download all videos' task out of our pods and into an AWS serverless system like Fargate (Lambda can only be used for shorter-running tasks). This has the advantages of (1) off-loading the tasks from our system, so that we don't have to worry about the consumption of our static resources, (2) autoscaling and parallel processing, (3) keeping the files in AWS (probably faster, more secure). The downside is that it is a more drastic change and would be more work to set up.
- Move the 'download all videos' task into its own queue, separate from the two other 'build' tasks. This would allow us to set the task concurrency to 1 for video zip tasks, with a higher value (4?) for the other build tasks, since those are shorter and more predictable. (Note that these queues would still share resources).
- Put the separate 'download all videos' task queue into its own pod/container, which would allow us to isolate its resources. We could also add autoscaling. This would be a larger change to our architecture but the task itself would be pretty much the same.
- Add prioritization to our tasks, so that shorter jobs can be moved up ahead of 'download all videos' tasks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the 'download all videos' task and the builder queue configuration; the issue does not name specific files or tests. Compare the proposed queue, pod, serverless, and prioritization approaches, then define and validate an architecture that prevents long video downloads from blocking EFP builds and data downloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.