alibaba / alibaba/SREWorks

[平台后续发展][DataOps][数智]任务调度及管理

Open
#130 1 comment 0 reactions 0 assignees View on GitHub
area/dataops type/design
Dominant language
Java
Stars
2k
Forks
430
PR merge metrics
No merged PRs in 30d

Description

SREWorks多次提到运维方面的数仓和ETL,并且也有Flink这类应用集成,后续有没有规划在这方面深入,如集成企业业务数仓相关功能?
1)数仓案例:一个数仓,采用Flink CDC做数据抽取,Flink SQL(批处理或流处理)或数据库SQL(批处理,如Hologres SQL)做数据处理,相关任务是需要上线的,批处理任务涉及到定时调度,流处理任务涉及到状态监控,任务可能会失败报错,需要收集错误日志,以上也属于运维的范畴,有没有计划增加Flink SQL和数据库SQL的任务调度及管理功能(参考Airflow、DolphinScheduler)?
2)数据资产:有没有计划通过Flink Catalog等方式收集各数据源信息来做一个这方面的元数据管理,进而实现数据资产目录和数据血缘功能(会用到上述的数据调度任务依赖元信息)。
3)数据服务:用户只需要写SQL,平台生成相关的数据服务API,以供外部访问数据。

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing the existing Flink integration and task-management capabilities, then compare them with the requested scheduling, metadata, lineage, and SQL-to-API scope. Done would require a defined, agreed feature scope before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, devops, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.