Reading big texts cause OutOfMemmory Error. How to read text partialy?

 I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

 

  Schema avroProj = SchemaBuilder.builder()

    .record("proj").fields()

    .name("idx").type().nullable().longType().noDefault()

    .name("text").type().nullable().bytesType().noDefault()

    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);

  ParquetReader parquetReader = AvroParquetReader

    .builder(new Path(filePath))

    .withConf(conf)

    .build();

  GenericRecord record = parquetReader.read(); // record already have full text in RAM

  Long idx = (Long) record.get("idx");

  ByteBuffer rawText = (ByteBuffer) record.get("text");

  String header = new String(rawText.array()).substring(0, 200);

**Environment**: I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

 
{code:java}
  Schema avroProj = SchemaBuilder.builder()
    .record("proj").fields()
    .name("idx").type().nullable().longType().noDefault()
    .name("text").type().nullable().bytesType().noDefault()
    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);
  ParquetReader parquetReader = AvroParquetReader
    .builder(new Path(filePath))
    .withConf(conf)
    .build();

  GenericRecord record = parquetReader.read();
// record already have full text in RAM
  Long idx = (Long) record.get("idx");
  ByteBuffer rawText = (ByteBuffer) record.get("text");
  String header = new String(rawText.array()).substring(0, 200);
{code}
**Reporter**: [Andrei Iatsuk](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=DrEwan)

_{**Note**: *This issue was originally created as [PARQUET-1237](https://issues.apache.org/jira/browse/PARQUET-1237). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

Beginne mit der im Issue gezeigten Projektion von AvroReadSupport und der Verwendung von AvroParquetReader und untersuche anschließend den Lesepfad für das Feld bytes. Als erledigt gilt die Bereitstellung einer unterstützten Möglichkeit, nur ein Textpräfix abzurufen, ohne jeden vollständigen Wert mit 15 MB zu materialisieren, einschließlich einer Abdeckung für den gemeldeten Out-of-Memory-Fall.

Vom Indexierungsmodell aus dem Issue-Text verfasst.