[平台后续发展][DataOps][数智]任务调度及管理
- Dominant language
- Java
- Stars
- 2k
- Forks
- 430
- PR merge metrics
- No merged PRs in 30d
Description
SREWorks多次提到运维方面的数仓和ETL,并且也有Flink这类应用集成,后续有没有规划在这方面深入,如集成企业业务数仓相关功能?
1)数仓案例:一个数仓,采用Flink CDC做数据抽取,Flink SQL(批处理或流处理)或数据库SQL(批处理,如Hologres SQL)做数据处理,相关任务是需要上线的,批处理任务涉及到定时调度,流处理任务涉及到状态监控,任务可能会失败报错,需要收集错误日志,以上也属于运维的范畴,有没有计划增加Flink SQL和数据库SQL的任务调度及管理功能(参考Airflow、DolphinScheduler)?
2)数据资产:有没有计划通过Flink Catalog等方式收集各数据源信息来做一个这方面的元数据管理,进而实现数据资产目录和数据血缘功能(会用到上述的数据调度任务依赖元信息)。
3)数据服务:用户只需要写SQL,平台生成相关的数据服务API,以供外部访问数据。
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing the existing Flink integration and task-management capabilities, then compare them with the requested scheduling, metadata, lineage, and SQL-to-API scope. Done would require a defined, agreed feature scope before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data-engineering, devops, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100