Reading big texts cause OutOfMemmory Error. How to read text partialy?

 I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

 

  Schema avroProj = SchemaBuilder.builder()

    .record("proj").fields()

    .name("idx").type().nullable().longType().noDefault()

    .name("text").type().nullable().bytesType().noDefault()

    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);

  ParquetReader parquetReader = AvroParquetReader

    .builder(new Path(filePath))

    .withConf(conf)

    .build();

  GenericRecord record = parquetReader.read(); // record already have full text in RAM

  Long idx = (Long) record.get("idx");

  ByteBuffer rawText = (ByteBuffer) record.get("text");

  String header = new String(rawText.array()).substring(0, 200);

**Environment**: I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

 
{code:java}
  Schema avroProj = SchemaBuilder.builder()
    .record("proj").fields()
    .name("idx").type().nullable().longType().noDefault()
    .name("text").type().nullable().bytesType().noDefault()
    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);
  ParquetReader parquetReader = AvroParquetReader
    .builder(new Path(filePath))
    .withConf(conf)
    .build();

  GenericRecord record = parquetReader.read();
// record already have full text in RAM
  Long idx = (Long) record.get("idx");
  ByteBuffer rawText = (ByteBuffer) record.get("text");
  String header = new String(rawText.array()).substring(0, 200);
{code}
**Reporter**: [Andrei Iatsuk](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=DrEwan)

_{**Note**: *This issue was originally created as [PARQUET-1237](https://issues.apache.org/jira/browse/PARQUET-1237). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

Issue に示されている AvroReadSupport の projection と AvroParquetReader の使用方法から始め、続いて bytes フィールドの reader path を調査します。各 15 MB の値全体を materialize せずにテキストの prefix だけを取得できる、サポートされた方法を提供し、報告された out-of-memory ケースをカバーできれば完了です。

索引モデルが issue の本文から書いたものです。