algolia / algolia/brefsearch-data

Heatmap retrieval can't be automated

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
2
Forks
1
PR merge metrics
No merged PRs in 30d

Description

The `heatmap` is used to represent the most replayed moments of a given video. I use it to pick which specific moment of a video is the most relevant, is several matches.

The heatmap data is not available through the YouTube API, only through `yt-dlp` (and even then, not for age-restricted videos). And `yt-dlp` doesn't work when run from GitHub Actions, as YouTube blocks the request, considering it comes from a bot.

Currently, I have set the heatmap update into its own `yarn run update:heatmap` script, that is expected to be run locally. The data rarely changes, so I assume it's ok.

But in the future, if we want to automatically add new videos, we might want to look at alternatives. I haven't tried them but maybe:

- We could use the --cookies options of `yt-dlp`, and read from Chrome/Firefox cookies. This is brittle as Cookies expires after a while, so would need to be regenerated, which means we'll need another system to use a Chrome headless to login to YouTube and save the Cookies somewhere for the main script to use. Doable, but probably messy.
- https://github.com/Benjamin-Loison/YouTube-operational-API seems to be an alternative YouTube API that does include the heatmap. There is no public API endpoint to test it (it has been taken down by YouTube which somehow vouch for its quality), but the repo can be cloned and installed. Maybe we can deploy it on fly.io and request it when we need it? Or maybe it will be plagued with the same bot restrictions. Only one way to find out...
- https://github.com/TarunTomar122/most-replayed-part-of-video seems to include code and video explaining how to get that data.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the `yarn run update:heatmap` script and its local data flow, then investigate the `yt-dlp` cookie approach and the two linked alternatives. The issue does not name source files, tests, or a settled implementation; done would require selecting and validating a reliable automated retrieval path.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, javascript
Domain
data, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.