aboutcode-org / aboutcode-org/purldb

Update checkpointing in alpine and maven pipelines

Aperta
#740 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
HTML
Stelle
67
Fork
69
Merge medio
8g 8h
PR unite (30g)
1

Descrizione

When running the alpine and maven pipelines to collect package urls, both actions time out after 6 hours. We should have a way to keep track of where we are when we are processing an index file so when we are interrupted, we can start off where we were.

For maven, the order of packages in the index is deterministic for a given version of the index, when the index is updated, the order of packages is not always the same as the order it was in the previous index.

We would need to keep track of the index file checksum to see if we can resume from the index file or if we have to start over because the index is different. We would also need to track which package we last processed so we know where to pick up from.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start by locating the alpine and maven pipeline entry points and the code that processes each index file. Check how the current index checksum and last processed package could be stored and restored. Done means an interrupted run resumes when the checksum matches and starts over when the index changes, for both pipelines.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Ambito
data-engineering
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
42/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.