Document recommendations for block size and page size given an expected number of writers

I sent a mail on dev list but I seem to have a problem with email on dev list so opening a bug here .

I am on a multithreaded system where there are M threads , each thread creating an independent parquet writer and writing on the hdfs in its own independent files . I have a finite amount of RAM say R .

Now when I created parquet writer using default block and page size i get heap error (no memory ) on my set up . so I reduced my block size and page size to very low and my system stopped giving me these out of memory errors and started writing the file correctly . I am able to read these files correctly as well .

I should not have to make the memory low and parquet should automatically make sure i do not get these errors .

But in case i have to keep track of the memory my question is as follows.

Now keeping these values very less is not a recommended practice as i would loose on performance . I am particularly concerned about write performance . What math formula do you recommend that I should use to find correct blockSize , pageSize to be passed to the parquet constructor to have the right WRITE performance . ie how can i decide what should be the right blockSize , pageSize for a parquet writer given that i have M threads and total RAM memory available is R . I don't understand dictionaryPageSize need and in case i need to bother about that as well kindly let me know but i have kept enableDictionary flag as false .

I am using the bellow constructor .
public More ...ParquetWriter(
162 Path file,
163 WriteSupport writeSupport,
164 CompressionCodecName compressionCodecName,
165 int blockSize,
166 int pageSize,
167 int dictionaryPageSize,
168 boolean enableDictionary,
169 boolean validating,
170 WriterVersion writerVersion,
171 Configuration conf) throws IOException {

**Reporter**: [Manish Agarwal](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=manish.agarwal)

_{**Note**: *This issue was originally created as [PARQUET-156](https://issues.apache.org/jira/browse/PARQUET-156). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

从 ParquetWriter 构造函数及其 blockSize、pageSize、dictionaryPageSize 和 enableDictionary 参数开始。确定项目能够针对 M 个 writer 和总内存 R 提供哪些指导，然后记录其中的权衡以及字典页相关的注意事项。完成的标准是用户能够获得以写入性能为重点、可直接执行的大小配置建议。

由索引模型根据 Issue 内容生成。