hdfswriter在win下使用idea调试出现的bug
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
### json文件如下
`{
"setting": {},
"job": {
"setting": {
"speed": {
"channel": 2
}
},
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": [
"vid",
"plane_no",
"ao_no",
"ao_name",
"work_station",
"stand_station",
"type",
"qty",
"stuff_code",
"re_stuff_code",
"tooling_number",
"tooling_qty",
"total_tooling_qty"
],
"splitPk": "vid",
"connection": [
{
"table": [
"tr_aoaao_resource"
],
"jdbcUrl": [
"jdbc:mysql://172.16.26.252:3306/test?useUnicode=true&characterEncoding=utf8"
]
}
]
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"defaultFS": "hdfs://172.16.26.51:9820",
"fileType": "orc",
"path": "/user/hive/warehouse/zishi_test.db/adm_tr_aoaao_resource_copy3",
"fileName": "adm_tr_aoaao_resource",
"column": [
{
"name": "vid",
"type": "BIGINT"
},
{
"name": "plane_no",
"type": "string"
},
{
"name": "work_station",
"type": "string"
},
{
"name": "stand_station",
"type": "string"
},
{
"name": "ao_no",
"type": "string"
},
{
"name": "ao_name",
"type": "string"
},
{
"name": "type",
"type": "string"
},
{
"name": "part_no",
"type": "string"
},
{
"name": "part_name",
"type": "string"
},
{
"name": "qty",
"type": "string"
},
{
"name": "stuff_code",
"type": "string"
},
{
"name": "re_stuff_code",
"type": "string"
},
{
"name": "tooling_number",
"type": "string"
},
{
"name": "tooling_qty",
"type": "string"
},
{
"name": "total_tooling_qty",
"type": "string"
}
],
"writeMode": "append",
"fieldDelimiter": "\u0001",
"compress":"NONE"
}
}
}
]
}
}`
最后输入的几行日志如下:
2021-04-06 15:05:39.266 INFO 5968 --- [ task-1] c.s.dataxofflinetask.task.OfflineTask : 2021-04-06 15:05:39.265 [job-0] INFO HdfsWriter$Job - start delete tmp dir [hdfs://172.16.23.51:9820/user/hive/warehouse/sfdb_dev.db] .
2021-04-06 15:05:39.421 INFO 5968 --- [ task-1] c.s.dataxofflinetask.task.OfflineTask : 2021-04-06 15:05:39.421 [job-0] INFO HdfsWriter$Job - finish delete tmp dir [hdfs://172.16.23.51:9820/user/hive/warehouse/sfdb_dev.db] .
**使用idea调试,把整个数据库删除了**
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the HdfsWriter$Job handling of temporary-directory deletion and the supplied hdfswriter configuration, then reproduce the Windows/IntelliJ run using the logged delete path. Determine why the deletion targets the database directory; done means the job no longer removes unrelated database data and the behavior is covered by an appropriate regression check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100