alibaba / alibaba/x-deeplearning
从kafka读数据训练怎么指定topic
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
关于从kafka读数据进行训练,文档上是这么写的:

结合data_reader.py代码,xxx部分应该是namenode,也就是broker list,那么topic在哪里指定呢。
看data_io.cc代码,应该是通过AddPath接口指定的,但是data_reader.py的_decode_path,并没有解析出fpath,因此AddPath的传递的也是个空,而DataSharding的代码看着也不支持对kafka数据分片。很疑惑,这块是不是少写了代码。请指教。
def _decode_path(self, path):
'''
hdfs://namenode/path
kafka://namenode
'''
namenode = ""
fs_type = None
fpath = None
if path.startswith('hdfs://'):
fs_type = pybind.fs.hdfs
arr = path.split('/', 3)
assert len(arr) == 4
namenode = arr[2]
fpath = '/'+arr[3]
elif path.startswith('kafka://'):
fs_type = pybind.fs.kafka
arr = path.split('/', 2)
assert len(arr) == 3
namenode = arr[2]
else:
assert '://' not in path, "Unsupported path: %s" % path
fpath = pathreturn fs_type, namenode, fpath
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with data_reader.py, especially _decode_path, then trace the AddPath call in data_io.cc and the DataSharding handling for Kafka paths. Verify how the broker list and topic are expected to be supplied; the issue is resolved when topic selection is supported or the documented configuration clearly matches the implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, kafka, python
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100