ClickHouse / ClickHouse/libhdfs3

hdfsRead() request more data then needed, so socket input buffer overflow

Open
#74 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
43
Forks
62
PR merge metrics
No merged PRs in 30d

Description

The hdfsRead() requests more data from datanode then needed, so TCP socket input buffer is overflow.

Calls chain: hdfsRead() -> InputStreamImpl::read() - > InputStreamImpl::readInternal() - > InputStreamImpl::readOneBlock() -> InputStreamImpl::setupBlockReader(bool) -> ctor RemoteBlockReader() with wrong 'len' argument
The length calculated here regardless requested data length:
https://github.com/ClickHouse/libhdfs3/blob/ceb428c52e6b4362a35ec18b69206d9bb94edce3/src/client/InputStreamImpl.cpp#L389
So RemoteBlockReader requests more data from a datanode which continue to transmit data until overflow message received. During this time a client can receive a lot of data (up to hundreds of megabytes for several connections) that occupy system memory while the socket is open.
If running in container, it can be seen in field 'sock' from /sys/fs/cgroup/memory.stat .
For example: read the row batches from parquet file in parallel mode, i.e. read a part of hdfs block, not the whole block in one call.

The hdfsRead() is used in ClickHouse here:
https://github.com/ClickHouse/ClickHouse/blob/34074c00b11245eebb45cdac98d4959107351b0d/src/Storages/ObjectStorage/HDFS/ReadBufferFromHDFS.cpp#L119
There is a "enable_hdfs_pread" config parameter ( default is true ). It enables using the hdfsPread call, which does not have this bug. But I'm not sure that hdfsRead is never called in this case.

Workaround: use hdfsPread() instead of hdfsRead().

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.