alibaba / alibaba/x-deeplearning
DataReader读本地目录数据出错
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
在跑deep_ctr例子的时候,将xdl.DataReader的path指定为本地目录:
`reader = xdl.DataReader("r1", # name of reader
paths=["/xdl-src/xdl_data/"], # file paths
enable_state=False) # enable reader state`
结果运行出错(完整的数据文件路径是/xdl-src/xdl_data/lr_data):
`root@50da243fe369:/xdl-src/xdl/examples/deepctr# python lr_model.py --run_mode=local
2019-01-11 10:30:40.875078: /xdl-compile/xdl/xdl/data_io/fs/file_system_local.cc:105] Check failed: lstat(path, &stat) != -1 can't stat lr_data
Aborted (core dumped)
`
看了下代码,有两个地方感觉写的有点问题:
1)FileSystemLocal::Dir里面对readdir的返回结果应该过滤当前目录(.)和父目录(..),当前代码如下
`while ((dirp = readdir(dp)) != nullptr) {
paths.push_back(dirp->d_name);
}
`
2)data_sharding里add_path的时候,保存的是相对路径, 导致lstat的时候找不到文件
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the deep_ctr example with lr_model.py and the local path shown in the issue. Read xdl/data_io/fs/file_system_local.cc, especially FileSystemLocal::Dir, then trace data_sharding's add_path handling. Done means local-directory reads no longer fail at lstat and directory traversal ignores the current and parent entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100