Reading big texts cause OutOfMemmory Error. How to read text partialy?

 I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

 

  Schema avroProj = SchemaBuilder.builder()

    .record("proj").fields()

    .name("idx").type().nullable().longType().noDefault()

    .name("text").type().nullable().bytesType().noDefault()

    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);

  ParquetReader parquetReader = AvroParquetReader

    .builder(new Path(filePath))

    .withConf(conf)

    .build();

  GenericRecord record = parquetReader.read(); // record already have full text in RAM

  Long idx = (Long) record.get("idx");

  ByteBuffer rawText = (ByteBuffer) record.get("text");

  String header = new String(rawText.array()).substring(0, 200);

**Environment**: I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

 
{code:java}
  Schema avroProj = SchemaBuilder.builder()
    .record("proj").fields()
    .name("idx").type().nullable().longType().noDefault()
    .name("text").type().nullable().bytesType().noDefault()
    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);
  ParquetReader parquetReader = AvroParquetReader
    .builder(new Path(filePath))
    .withConf(conf)
    .build();

  GenericRecord record = parquetReader.read();
// record already have full text in RAM
  Long idx = (Long) record.get("idx");
  ByteBuffer rawText = (ByteBuffer) record.get("text");
  String header = new String(rawText.array()).substring(0, 200);
{code}
**Reporter**: [Andrei Iatsuk](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=DrEwan)

_{**Note**: *This issue was originally created as [PARQUET-1237](https://issues.apache.org/jira/browse/PARQUET-1237). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

Inizia con la proiezione di AvroReadSupport e l'utilizzo di AvroParquetReader mostrati nell'issue, quindi analizza il percorso di lettura del campo bytes. Il lavoro è completato quando esiste un modo supportato per ottenere solo un prefisso di testo senza materializzare ogni valore completo da 15 MB, con una copertura per il caso di esaurimento della memoria segnalato.

Scritto dal modello di indicizzazione a partire dal testo della issue.