apache / apache/arrow-go

[Go] Optimize memory usage for CSV writer

Open
#78 0 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

perhaps not for this PR, but, depending on the number of rows and cols this record contains, this may be a very large allocation, and very big memory chunk.
it could be more interesting performance wise to write n rows instead of everything in one big chunk.

also, to reduce the memory pressure on the GC, we should probably (re)use this slice-of-slice of strings.

 

**Reporter**: [Anson Qian](https://issues.apache.org/jira/browse/ARROW-4736) / @anson627

**Note**: *This issue was originally created as [ARROW-4736](https://issues.apache.org/jira/browse/ARROW-4736). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.