Document recommendations for block size and page size given an expected number of writers

I sent a mail on dev list but I seem to have a problem with email on dev list so opening a bug here .

I am on a multithreaded system where there are M threads , each thread creating an independent parquet writer and writing on the hdfs in its own independent files . I have a finite amount of RAM say R .

Now when I created parquet writer using default block and page size i get heap error (no memory ) on my set up . so I reduced my block size and page size to very low and my system stopped giving me these out of memory errors and started writing the file correctly . I am able to read these files correctly as well .

I should not have to make the memory low and parquet should automatically make sure i do not get these errors .

But in case i have to keep track of the memory my question is as follows.

Now keeping these values very less is not a recommended practice as i would loose on performance . I am particularly concerned about write performance . What math formula do you recommend that I should use to find correct blockSize , pageSize to be passed to the parquet constructor to have the right WRITE performance . ie how can i decide what should be the right blockSize , pageSize for a parquet writer given that i have M threads and total RAM memory available is R . I don't understand dictionaryPageSize need and in case i need to bother about that as well kindly let me know but i have kept enableDictionary flag as false .

I am using the bellow constructor .
public More ...ParquetWriter(
162 Path file,
163 WriteSupport writeSupport,
164 CompressionCodecName compressionCodecName,
165 int blockSize,
166 int pageSize,
167 int dictionaryPageSize,
168 boolean enableDictionary,
169 boolean validating,
170 WriterVersion writerVersion,
171 Configuration conf) throws IOException {

**Reporter**: [Manish Agarwal](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=manish.agarwal)

_{**Note**: *This issue was originally created as [PARQUET-156](https://issues.apache.org/jira/browse/PARQUET-156). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

Bắt đầu với constructor của ParquetWriter và các tham số blockSize, pageSize, dictionaryPageSize và enableDictionary. Xác định những hướng dẫn nào dự án có thể hỗ trợ cho M writer và tổng bộ nhớ R, sau đó ghi lại các đánh đổi và những điểm cần cân nhắc về dictionary page. Được xem là hoàn tất khi người dùng có các khuyến nghị định cỡ có thể áp dụng, tập trung vào hiệu năng ghi.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.