Optimization - Create Listen from Lastfm Listen / Separate Queue for Track Names Not Yet Researched

# Background
Every time a user listens to a song, we receive a `Last.fm Listen` which is not linked to a Spotify Track.

## Friction - Backfilling a new listener's history can take over 1 day
The system can keep up with the daily listens for the 50 listeners its tracking right now, but backfilling a new listener takes a very long time.

When a new listener is added, we ingest their entire listening history. This could be up to 100,000 Listens.

Right now, we can link roughly 720 unique tracks per hour. Depending on how much overlap is in a listener's history (or overlap with other new listeners being backfilled), that translates to 1k-40k events per hour. At that rate, backfilling a single listener can take over 1 day.

## Bottleneck - Spotify API Calls
Our music service picks these listening events up one-by-one and performs the following steps:

1) Identify Matching Track
a) Search for Exact Match by Track Name and Artist Name
b) Search for a Previous Fuzzy Match*
c) Search Spotify using Track Name and Artist Name**
2) Link All not-yet-researched Last.fm Listens with the same Track Name and Artist Name to This Track

*(i.e. have we previously linked a `Last.fm Listen` with this same track name and artist name)
**(this is where fuzzy matches from from, Spotify's judgement)

### Step 1.c (_Search Spotify_) is the bottleneck
Spotify's rate limit is low and sometimes unpredictable (see Appendix). When we get rate limited, we get rate limited very hard (e.g. wait >16 hours). To avoid this, we run the entire system very conservatively (i.e. the 720 req/hour).

# Approaches
## Approach - Identify Listen Events that Require Spotify and Process Them in a Separate Queue
Right now, we throttle the whole system based on Spotify's rate limit. So, even if most of the listening events are getting matched in our own database, we still keep the system running slowly. This avoids the risk of getting throttled by Spotify, but many listening events that could be process immediately must wait.

### Variant - Put Listen Events that Require Spotify in a queue
_Identify Matching Track Internally_
1) Search Exact Match
2) Search Previous Fuzzy Match
3) Link or Defer
- Condition - Track Found
- Link all Pending, Matching Listening Events
- Condition - Track Not Found
- Put in Search Spotify Queue

_Search Spotify_
1) Pull from Search Spotify Queue
2) Search Spotify
3) Link Track
- Condition - Track Found
- Link all Pending, Matching Listening Events
- Condition - Track Not Found
- Mark All Pending, Matching Listening Events as Analyzed

#### Storage Options
- AWS SQS
- In-memory Queue
- On the Listens Table (e.g. add a status)

## Approach - Profile Spotify's Rate Limit and Increase Throughput to that Limit
The 720 requests per hour is a conservative limit. We could likely increase the speed of the entire system up to the actual limit, but we've found Spotify's limit to be unpredictable and inconsistent with its documentation (see appendix).

We could profile Spotify to find the actual limit.

- ▲ Pro - Simple, does not increase the complexity of the system
- ▼ Con - Not resilient to a change (documented or not documented) in Spotify's rate limit policy
- ▼ Con - Does not speed up the ability to link tracks with information in the database

# ➡️ Proposal - Process Spotify Events in a Separate Queue; Then determine if increasing Spotify processing speed is required
Isolating listening events that require Spotify...

- ...completely removes the restriction of Spotify api calls on events that can be linked with our own data
- 📍 Position - Adam: The value of this will grow significantly as our own dataset grows to cover more and more of Spotify
- ...isolates Spotify events into a queue that can be further optimized

# Appendix
## Spotify's Rate Limit doesn't seem to match its documentation
Resource - [Spotify's Documentation - Rate Limits](https://developer.spotify.com/documentation/web-api/concepts/rate-limits)

- Claim - Spotify considers a rolling 30 second window when considering the rate limit
- 👀 Observation - Spotify will let us make many, many requests for a long time before throttling us. When it does throttle us, it's `retry_after` header is very large (e.g. 16 hours)

# Questions
- ❓ Question - Is fixing this worthwhile?
- ❓ Question - During a backfills, what portion of listen events do not require Spotify?