googleapis / googleapis/google-api-java-client

Improve memory efficiency of base64-encoded byte array decoding operations

Abierto
#967 4 comentarios 0 reacciones 0 asignados Ver en GitHub
priority: p4 type: feature request
Lenguaje dominante
Java
Estrellas
1.5k
Forks
772
Merge medio
1 h 44 min
PR fusionados (30 d)
1

Descripción

Currently, decoding base64-encoded byte array with the Java API client is very inefficient in terms of memory usage.

Reading big attachments from the GMail API in memory-constrained environments like App Engine is quite a challenge because of that.
https://developers.google.com/gmail/api/v1/reference/users/messages/attachments/get

On a 64 bit Java 7 VM, it required at least 115 MB of heap (-Xmx115M) to read a 12Mb attachment from the Gmail API (which is a 17MB base64 string in the response from the API).

The code to test it is dead simple (MESSAGE_ID / ATTACHMENT_ID references a big attachment):

``` java
gmail.users().messages().attachments().get("me", MESSAGE_ID, ATTACHMENT_ID).execute()
```

Here is the stacktrace when trying to load the attachment with only 110MB or heap:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:2694)
at java.lang.String.(String.java:203)
at java.lang.StringBuilder.toString(StringBuilder.java:405)
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:360)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:277)
at com.google.api.client.json.jackson2.JacksonParser.getText(JacksonParser.java:76)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:850)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:471)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:780)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:381)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:354)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:87)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:81)
at com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:459)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at LoadEmail.main(LoadEmail.java:31)

Obtaining a stream from the data field in the response would help keep the memory usage lower.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.