alibaba / alibaba/DataX

微软云上mysql同步到hive时报错

Open
#679 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

使用datax进行同步数据到hive时,主要有2个比较重要的参数:
"defaultFS"(值一般是 hdfs://namenode:8020)
"path"(hive表对应的hdfs文件路径,如 /user/hadoop01/tmp/table_001

我们这边根据实际情况配置如下,
"defaultFS": "hdfs://10.0.3.18:8020",
"path": "/user/hadoop01/tmp/table_001",

执行时报错如下:
com.alibaba.datax.common.exception.DataXException: Code:[HdfsWriter-02], Description:[您填写的参数值不合法.]. - 您配置的path: [/user/hadoop01/tmp/table_001] 不存在, 请先在hive端创建对应的数据库和表.

换成wasb开头的配置也试过:
"defaultFS": "wasb://bigdata04-2020-05-15t02-04-27-aaaa@bigdata04111hdistorage.blob.core.windows.net",
"path": "wasb://bigdata04-2020-05-15t02-04-27-aaaa@bigdata04111hdistorage.blob.core.windows.net/user/hadoop01/tmp/table_001",

hadoop识别不了wasb,执行时报错如下:
Caused by: java.io.IOException: No FileSystem for scheme: wasb
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2644)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)

有人遇到类似问题吗

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the DataX HdfsWriter-02 validation that reports the missing path, then follow the Hadoop FileSystem stack trace for the unsupported wasb scheme. Reproduce the MySQL-to-Hive configuration shown in the issue and determine whether the failure is in DataX configuration or Hadoop filesystem support. Done means an actionable, verified configuration or a clearly documented compatibility limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, hadoop, java, mysql
Domain
cloud, data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.