alibaba / alibaba/DataX

基于DataX 开发了一个 Pipeline AI Agent

Open
#2,341 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

## 介绍
本人基于DataX 开发了一个 Pipeline AI Agent,自动构建批量数据同步通道,只需要你编写一句自然语言就能把一个 基于DataX的数据通道构建起来,并且触发执行,目前处于阿尔法测试阶段,有兴趣的同学可以试试顺便帮提提意见

## 交互界面:

Image

## 任务描述例子

> 我有一个本地文件需要导入到mysql数据库中,文件路径为:/opt/misc/dfs/totalpayinfo__47a61b2a_7e96_409d_8769_0143423bebba.csv,文件格式为csv,mysql库使用名称为‘shop’的数据库源作为目标端导入对象

附带有目标端数据库配置信息
>我有一个本地文件需要导入到mysql数据库中,文件路径为:/opt/misc/dfs/totalpayinfo__47a61b2a_7e96_409d_8769_0143423bebba.csv,文件格式为csv,mysql 数据库host为:192.168.28.200,端口为3306,用户名为root,密码为123456,dbname为:shop

mysql同步到mysql
>创建一个管道从mysql同步到mysql,实例名称为:mysql-mysql-order。源端数据库配置是 host:192.168.28.200 ,端口:3306,dbname:order2,用户名:root,密码为:123456,目标端配置,host:192.168.28.200,端口:3306,dbname:shop ,用户名:root,密码为123456。从目标端选取几张表,需要表前缀为:order

kingbase同步到mysql
> 构建一个kingbase同步到mysql的数据管道,需要创建实时同步,需要选取表为flow_node_expand,kingbase的配置为:host: 192.168.28.201,端口:4321,dbname:test,用户名:kingbase,密码:123456,schema:public,kingbase使用mysql兼容数据库类型

mysql到kafka的
> 创建mysql到kafka的数据的数据同步管道,mysql的配置使用名称为order2的数据源,使用其中的 orderdetail表,kafka的配置为,服务端地址为:192.168.28.201:9092,topic为test

## 相关文档:
1. 功能说明:https://tis.pub/docs/guide/pipeline-ai-agent/
2. 项目官网:https://github.com/datavane/tis

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository files, tests, or entry points are named. Start by reading the linked pipeline AI agent documentation and comparing it with DataX's current scope; the issue does not define a repository change or acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, mysql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.