Automatically allocate a new chunk group instead of throwing error due to buffer size limits
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
As in #5918, we might throw an error when writing into columnar storage due to several buffer size limits enforced by safestring library or by Postgres.
While the buffer size limit for `memcpy_s()` is 256MB, `memcpy()` doesn't enforce such a limitation; so #6419 attempts improving this situation by switching to use plain `memcpy()` instead `memcpy_s()`.
Note that even if we use `memcpy()`, memory allocators --such as `enlargeStringInfo()`-- still enforces a limit of 1GB when writing a chunk group into the disk.
Indeed, we could almost completely remove such memory limitations by deciding to allocate a new chunk group in `ColumnarWriteRow()` instead of throwing an error in the runtime (i.e., when trying to expand the latest chunk group based on GUC limits).
Contributor guide
Assessment
This issue has not been assessed yet.