Reading big texts cause OutOfMemmory Error. How to read text partialy?

 I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

  Schema avroProj = SchemaBuilder.builder()

    .record("proj").fields()

    .name("idx").type().nullable().longType().noDefault()

    .name("text").type().nullable().bytesType().noDefault()

    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);

  ParquetReader parquetReader = AvroParquetReader

    .builder(new Path(filePath))

    .withConf(conf)

    .build();

  GenericRecord record = parquetReader.read(); // record already have full text in RAM

  Long idx = (Long) record.get("idx");

  ByteBuffer rawText = (ByteBuffer) record.get("text");

  String header = new String(rawText.array()).substring(0, 200);

Environment: I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

 
{code:java}
  Schema avroProj = SchemaBuilder.builder()
    .record("proj").fields()
    .name("idx").type().nullable().longType().noDefault()
    .name("text").type().nullable().bytesType().noDefault()
    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);
  ParquetReader parquetReader = AvroParquetReader
    .builder(new Path(filePath))
    .withConf(conf)
    .build();

  GenericRecord record = parquetReader.read();
// record already have full text in RAM
  Long idx = (Long) record.get("idx");
  ByteBuffer rawText = (ByteBuffer) record.get("text");
  String header = new String(rawText.array()).substring(0, 200);
{code}
Reporter: Andrei Iatsuk

_{Note: This issue was originally created as PARQUET-1237. Please see the migration documentation for further details.}

Start with the AvroReadSupport projection and AvroParquetReader usage shown in the issue, then investigate the reader path for the bytes field. Done means a supported way to obtain only a text prefix without materializing each full 15 MB value, with coverage for the reported out-of-memory case.