NVIDIA / NVIDIA/cudf

`CompactProtocolWriter` and `ProtobufWriter` API provide no encapsulation of the output buffer

Open
#7,015 3 comments 0 reactions 0 assignees View on GitHub
cuIO improvement libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Problem:**
The protocol writer classes take a pointer to a vector and use it as the output buffer. Writes change the size of this vector. This vector is also modified (including size changes) outside of the writers. The ORC/Parquet writers have a `std::vector` data member that is reused for protocol writes and manually reset between uses. ORC writer also reuses the `ProtobufWriter` object. In addition, Parquet writer reuses the output buffer to output data unrelated to `CompactProtocolWriter`. All this makes the use error-prone.

**Solution proposal:**
Modify the protocol writer API to use an internal output buffer and only provide getters for it. Also, protocol writer objects should not be reused (and cannot, with the proposed API). There shouldn't be a buffer data member in `xyz::writer::impl`.
These changes would limit the scope of the state to functions instead of the lifetime of `xyz::writer::impl` objects.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.