aboutcode-org / aboutcode-org/vulnerablecode

Get rid of the custom `paginated()` and use the built-in `iterator()`

Abierto
#1,798 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Priority: high
Lenguaje dominante
Python
Estrellas
702
Forks
328
Merge medio
3 d 8 h
PR fusionados (30 d)
3

Descripción

The custom `BaseQuerySet.paginated()` performs extremely poorly in terms of compute time and memory usage compared to the built-in `iterator()`.

## Below is an example where we compare `.paginated()` with `.iterator()` for iterating over 8 million advisory data.

### Using custom `paginated()`

Even after 20 minutes, the iteration is not completed, and the ETA keeps getting worse and worse.

```python
Iterating advisory data using custom BaseQuerySet.paginated()
INFO 2025-03-07 08:56:41.500 Pipeline [IterateAdvisory] starting
INFO 2025-03-07 08:56:41.500 Step [iterate_advisories] starting
INFO 2025-03-07 08:56:41.796 Iterating over 8073782 Advisory objects.
INFO 2025-03-07 08:57:50.106 Progress: 10% (807379/8073782) ETA: 615 seconds (10.2 minutes)
INFO 2025-03-07 09:00:09.226 Progress: 20% (1614757/8073782) ETA: 830 seconds (13.8 minutes)
INFO 2025-03-07 09:03:34.920 Progress: 30% (2422135/8073782) ETA: 964 seconds (16.1 minutes)
INFO 2025-03-07 09:09:04.444 Progress: 40% (3229513/8073782) ETA: 1114 seconds (18.6 minutes)
INFO 2025-03-07 09:16:32.571 Progress: 50% (4036891/8073782) ETA: 1191 seconds (19.9 minutes)
^CCommandError: Keyboard interrupt received. Stopping...
```

### Using built-in `iterator()`

It takes around 4 minutes to iterate over 8 million advisories.
```python
Iterating advisory data using built-in .iterator()
INFO 2025-03-07 10:02:43.483 Pipeline [IterateAdvisory] starting
INFO 2025-03-07 10:02:43.483 Step [iterate_advisories] starting
INFO 2025-03-07 10:02:44.042 Iterating over 8073782 Advisory objects.
INFO 2025-03-07 10:05:00.181 Progress: 10% (807379/8073782) ETA: 1225 seconds (20.4 minutes)
INFO 2025-03-07 10:05:19.790 Progress: 20% (1614757/8073782) ETA: 623 seconds (10.4 minutes)
INFO 2025-03-07 10:05:37.013 Progress: 30% (2422135/8073782) ETA: 404 seconds (6.7 minutes)
INFO 2025-03-07 10:05:50.865 Progress: 40% (3229513/8073782) ETA: 280 seconds (4.7 minutes)
INFO 2025-03-07 10:06:01.161 Progress: 50% (4036891/8073782) ETA: 197 seconds (3.3 minutes)
INFO 2025-03-07 10:06:11.465 Progress: 60% (4844270/8073782) ETA: 138 seconds (2.3 minutes)
INFO 2025-03-07 10:06:21.761 Progress: 70% (5651648/8073782) ETA: 93 seconds (1.6 minutes)
INFO 2025-03-07 10:06:32.129 Progress: 80% (6459026/8073782) ETA: 57 seconds
INFO 2025-03-07 10:06:42.248 Progress: 90% (7266404/8073782) ETA: 26 seconds
INFO 2025-03-07 10:06:52.325 Progress: 100% (8073782/8073782)
INFO 2025-03-07 10:06:52.587 Step [iterate_advisories] completed in 249 seconds (4.2 minutes)
INFO 2025-03-07 10:06:52.587 Pipeline completed in 249 seconds (4.2 minutes)
```

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.