citusdata / citusdata/citus

Automatically allocate a new chunk group instead of throwing error due to buffer size limits

Open
#6,420 2 comments 1 reaction 0 assignees View on GitHub
columnar
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

As in #5918, we might throw an error when writing into columnar storage due to several buffer size limits enforced by safestring library or by Postgres.

While the buffer size limit for `memcpy_s()` is 256MB, `memcpy()` doesn't enforce such a limitation; so #6419 attempts improving this situation by switching to use plain `memcpy()` instead `memcpy_s()`.
Note that even if we use `memcpy()`, memory allocators --such as `enlargeStringInfo()`-- still enforces a limit of 1GB when writing a chunk group into the disk.

Indeed, we could almost completely remove such memory limitations by deciding to allocate a new chunk group in `ColumnarWriteRow()` instead of throwing an error in the runtime (i.e., when trying to expand the latest chunk group based on GUC limits).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.