alibaba / alibaba/x-deeplearning

从kafka读数据训练怎么指定topic

Open
#127 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

关于从kafka读数据进行训练,文档上是这么写的:
![image](https://user-images.githubusercontent.com/11550277/52691687-9607ad80-2f9c-11e9-8979-ce8e982dc5ee.png)
结合data_reader.py代码,xxx部分应该是namenode,也就是broker list,那么topic在哪里指定呢。
看data_io.cc代码,应该是通过AddPath接口指定的,但是data_reader.py的_decode_path,并没有解析出fpath,因此AddPath的传递的也是个空,而DataSharding的代码看着也不支持对kafka数据分片。很疑惑,这块是不是少写了代码。请指教。

    def _decode_path(self, path):

'''
hdfs://namenode/path
kafka://namenode
'''
namenode = ""
fs_type = None
fpath = None
if path.startswith('hdfs://'):
fs_type = pybind.fs.hdfs
arr = path.split('/', 3)
assert len(arr) == 4
namenode = arr[2]
fpath = '/'+arr[3]
elif path.startswith('kafka://'):
fs_type = pybind.fs.kafka
arr = path.split('/', 2)
assert len(arr) == 3
namenode = arr[2]
else:
assert '://' not in path, "Unsupported path: %s" % path
fpath = path

return fs_type, namenode, fpath

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with data_reader.py, especially _decode_path, then trace the AddPath call in data_io.cc and the DataSharding handling for Kafka paths. Verify how the broker list and topic are expected to be supplied; the issue is resolved when topic selection is supported or the documented configuration clearly matches the implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, kafka, python
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.