aboutcode-org / aboutcode-org/vulnerablecode

Find a scalable way to archive reference URLS

Abierto
#2,303 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
702
Forks
328
Merge medio
3 d 8 h
PR fusionados (30 d)
3

Descripción

As originally posted by @ziadhany in this issue, we now have a way to collect reference URLs, but there are issue with doing it at scale, including rate limiting:
- https://github.com/aboutcode-org/vulnerablecode/issues/1724#issuecomment-4110565625

> I think we should use web.archive.org to store archive URLs via their "Save Page Now" endpoint `https://web.archive.org/save/`
>
> https://github.com/akamhy/waybackpy/blob/master/waybackpy/save_api.py.
> Since our URLs are stored in AdvisoryReference, we could add a new field `archive_url` to that model.
>
> One major constraint is the rate limit of 15 requests per minute. With 250,000 URLs to process, the initial run would take about 11.5 days:
>
> 250,000 / (15 * 60 * 24) = 11.57 days

@ziadhany is reaching out to archive.org to find better and improved ways to archive at scale.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.