alibaba / alibaba/x-deeplearning

DataReader读本地目录数据出错

Open
#61 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

在跑deep_ctr例子的时候,将xdl.DataReader的path指定为本地目录:
`reader = xdl.DataReader("r1", # name of reader
paths=["/xdl-src/xdl_data/"], # file paths
enable_state=False) # enable reader state`
结果运行出错(完整的数据文件路径是/xdl-src/xdl_data/lr_data):
`root@50da243fe369:/xdl-src/xdl/examples/deepctr# python lr_model.py --run_mode=local
2019-01-11 10:30:40.875078: /xdl-compile/xdl/xdl/data_io/fs/file_system_local.cc:105] Check failed: lstat(path, &stat) != -1 can't stat lr_data
Aborted (core dumped)
`

看了下代码,有两个地方感觉写的有点问题:
1)FileSystemLocal::Dir里面对readdir的返回结果应该过滤当前目录(.)和父目录(..),当前代码如下
`while ((dirp = readdir(dp)) != nullptr) {
paths.push_back(dirp->d_name);
}
`
2)data_sharding里add_path的时候,保存的是相对路径, 导致lstat的时候找不到文件

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the deep_ctr example with lr_model.py and the local path shown in the issue. Read xdl/data_io/fs/file_system_local.cc, especially FileSystemLocal::Dir, then trace data_sharding's add_path handling. Done means local-directory reads no longer fail at lstat and directory traversal ignores the current and parent entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.