alibaba / alibaba/DataX

hdfswriter在win下使用idea调试出现的bug

Open
#990 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

### json文件如下
`{
"setting": {},
"job": {
"setting": {
"speed": {
"channel": 2
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"vid",
"plane_no",
"ao_no",
"ao_name",
"work_station",
"stand_station",
"type",
"qty",
"stuff_code",
"re_stuff_code",
"tooling_number",
"tooling_qty",
"total_tooling_qty"
],
"splitPk": "vid",
"connection": [
{
"table": [
"tr_aoaao_resource"
],
"jdbcUrl": [
"jdbc:mysql://172.16.26.252:3306/test?useUnicode=true&characterEncoding=utf8"
]
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://172.16.26.51:9820",
"fileType": "orc",
"path": "/user/hive/warehouse/zishi_test.db/adm_tr_aoaao_resource_copy3",
"fileName": "adm_tr_aoaao_resource",
"column": [
{
"name": "vid",
"type": "BIGINT"
},
{
"name": "plane_no",
"type": "string"
},
{
"name": "work_station",
"type": "string"
},
{
"name": "stand_station",
"type": "string"
},
{
"name": "ao_no",
"type": "string"
},
{
"name": "ao_name",
"type": "string"
},
{
"name": "type",
"type": "string"
},
{
"name": "part_no",
"type": "string"
},
{
"name": "part_name",
"type": "string"
},
{
"name": "qty",
"type": "string"
},
{
"name": "stuff_code",
"type": "string"
},
{
"name": "re_stuff_code",
"type": "string"
},
{
"name": "tooling_number",
"type": "string"
},
{
"name": "tooling_qty",
"type": "string"
},
{
"name": "total_tooling_qty",
"type": "string"
}
],
"writeMode": "append",
"fieldDelimiter": "\u0001",
"compress":"NONE"
}
}
}
]
}
}`

最后输入的几行日志如下:
2021-04-06 15:05:39.266 INFO 5968 --- [ task-1] c.s.dataxofflinetask.task.OfflineTask : 2021-04-06 15:05:39.265 [job-0] INFO HdfsWriter$Job - start delete tmp dir [hdfs://172.16.23.51:9820/user/hive/warehouse/sfdb_dev.db] .
2021-04-06 15:05:39.421 INFO 5968 --- [ task-1] c.s.dataxofflinetask.task.OfflineTask : 2021-04-06 15:05:39.421 [job-0] INFO HdfsWriter$Job - finish delete tmp dir [hdfs://172.16.23.51:9820/user/hive/warehouse/sfdb_dev.db] .

**使用idea调试,把整个数据库删除了**

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the HdfsWriter$Job handling of temporary-directory deletion and the supplied hdfswriter configuration, then reproduce the Windows/IntelliJ run using the logged delete path. Determine why the deletion targets the database directory; done means the job no longer removes unrelated database data and the behavior is covered by an appropriate regression check.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.