airbnb / airbnb/binaryalert

Create a different queue for retroactive analysis

Open
#134 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
179
PR merge metrics
No merged PRs in 30d

Description

## Background

Related to #102 , it's always felt awkward to have the same SQS queue for both regular operation and for retroactive analysis:

- New binaries added to the BinaryAlert bucket might not be scanned for hours if a large retro scan is underway
- A `live_test` will fail for the same reason if a retro scan is running
- A `purge_queue` to stop a runaway retro scan will also drop new object events from the queue
- The number of concurrent analyzers needed for a retro scan vs. normal operation could be vastly different

These problems could all be fixed if there were just a different SQS queue for retroactive analysis!

## Desired Change

Create a new SQS queue specifically for retro scans. It's not clear whether Lambda will allow 2 different SQS event source mappings, but it's worth a shot.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the existing SQS queue and the retroactive scan Lambda configuration, including the Terraform deployment settings. Check whether two SQS event source mappings are supported, then verify that retro scans use a separate queue without delaying live events or affecting queue purging. Done means the separate queue is deployable and both workflows operate independently.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, terraform
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.