Reading big texts cause OutOfMemmory Error. How to read text partialy?

 I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

 

  Schema avroProj = SchemaBuilder.builder()

    .record("proj").fields()

    .name("idx").type().nullable().longType().noDefault()

    .name("text").type().nullable().bytesType().noDefault()

    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);

  ParquetReader parquetReader = AvroParquetReader

    .builder(new Path(filePath))

    .withConf(conf)

    .build();

  GenericRecord record = parquetReader.read(); // record already have full text in RAM

  Long idx = (Long) record.get("idx");

  ByteBuffer rawText = (ByteBuffer) record.get("text");

  String header = new String(rawText.array()).substring(0, 200);

**Environment**: I have dataset with big strings (every record about 15 mb) in parquet.

When I try to open all parquet parts I get OutOfMemory exception.

How can I get only headers (first 100 symbols) for each string record without reading all record?

 
{code:java}
  Schema avroProj = SchemaBuilder.builder()
    .record("proj").fields()
    .name("idx").type().nullable().longType().noDefault()
    .name("text").type().nullable().bytesType().noDefault()
    .endRecord();

  Configuration conf = new Configuration();

  AvroReadSupport.setRequestedProjection(conf, avroProj);
  ParquetReader parquetReader = AvroParquetReader
    .builder(new Path(filePath))
    .withConf(conf)
    .build();

  GenericRecord record = parquetReader.read();
// record already have full text in RAM
  Long idx = (Long) record.get("idx");
  ByteBuffer rawText = (ByteBuffer) record.get("text");
  String header = new String(rawText.array()).substring(0, 200);
{code}
**Reporter**: [Andrei Iatsuk](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=DrEwan)

_{**Note**: *This issue was originally created as [PARQUET-1237](https://issues.apache.org/jira/browse/PARQUET-1237). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*}

Commencez par la projection de AvroReadSupport et l’utilisation de AvroParquetReader présentées dans l’issue, puis examinez le chemin de lecture du champ bytes. Le travail est terminé lorsqu’il existe un moyen pris en charge d’obtenir uniquement un préfixe de texte sans matérialiser chaque valeur complète de 15 Mo, avec une couverture du cas de manque de mémoire signalé.

Rédigé par le modèle d'indexation à partir du texte de l'issue.