allegro / allegro/bigflow

Metadata storage for jobs/workflows

Aperta
#240 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
120
Fork
23
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Metadata storage for bigflow jobs/workflows

There are several usecases for simple document/key-value storage

1. Save (append) information about executed workflows/jobs.
ID, run-time, docker hash, execution time, cost estimate, result etc...
Basically some sort of structured logs, which may be used to
see execution history & do some cost estimation (manually)

2. Query for running workflows/jobs, their status (history and/or curenly running workflows)
> bigflow history -w workflow_id
Such cli api migh be a first step towards "airflow-free" solution
(aka ability to replace airflow with custom cron-like service)

3. Communicate between taks/workflows.
In some rare cases one workflow migh want to check status of another.
Also workflow migh check if another instance is currently running.
This especially important for dev-like environments, where
workflows are executed locally (via bigflow run).

4. Persist some information between tasks/jobs.
Like 'last-processed-id' (for incremental processing),
last time-per-batch (to auto-adjust batch-size) etc.

Database - anything for 1. BigQuery / any-sql-like DB for 1/2/3/4.

Client visible API - TBD.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.