Metadata storage for jobs/workflows
- 主要語言
- Python
- 星號
- 120
- 分支
- 23
- PR 合併指標
- 30 天內沒有已合併 PR
描述
Metadata storage for bigflow jobs/workflows
There are several usecases for simple document/key-value storage
1. Save (append) information about executed workflows/jobs.
ID, run-time, docker hash, execution time, cost estimate, result etc...
Basically some sort of structured logs, which may be used to
see execution history & do some cost estimation (manually)
2. Query for running workflows/jobs, their status (history and/or curenly running workflows)
> bigflow history -w workflow_id
Such cli api migh be a first step towards "airflow-free" solution
(aka ability to replace airflow with custom cron-like service)
3. Communicate between taks/workflows.
In some rare cases one workflow migh want to check status of another.
Also workflow migh check if another instance is currently running.
This especially important for dev-like environments, where
workflows are executed locally (via bigflow run).
4. Persist some information between tasks/jobs.
Like 'last-processed-id' (for incremental processing),
last time-per-batch (to auto-adjust batch-size) etc.
Database - anything for 1. BigQuery / any-sql-like DB for 1/2/3/4.
Client visible API - TBD.
貢獻指南
這個儲存庫沒有索引到貢獻指南
評估
這個 Issue 還沒有評估資料。