googleapis / googleapis/google-api-java-client

Improve memory efficiency of base64-encoded byte array decoding operations

Aperta
#967 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
priority: p4 type: feature request
Lingua principale
Java
Stelle
1.5k
Fork
772
Merge medio
1h 44m
PR unite (30g)
1

Descrizione

Currently, decoding base64-encoded byte array with the Java API client is very inefficient in terms of memory usage.

Reading big attachments from the GMail API in memory-constrained environments like App Engine is quite a challenge because of that.
https://developers.google.com/gmail/api/v1/reference/users/messages/attachments/get

On a 64 bit Java 7 VM, it required at least 115 MB of heap (-Xmx115M) to read a 12Mb attachment from the Gmail API (which is a 17MB base64 string in the response from the API).

The code to test it is dead simple (MESSAGE_ID / ATTACHMENT_ID references a big attachment):

``` java
gmail.users().messages().attachments().get("me", MESSAGE_ID, ATTACHMENT_ID).execute()
```

Here is the stacktrace when trying to load the attachment with only 110MB or heap:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:2694)
at java.lang.String.(String.java:203)
at java.lang.StringBuilder.toString(StringBuilder.java:405)
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:360)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:277)
at com.google.api.client.json.jackson2.JacksonParser.getText(JacksonParser.java:76)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:850)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:471)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:780)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:381)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:354)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:87)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:81)
at com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:459)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at LoadEmail.main(LoadEmail.java:31)

Obtaining a stream from the data field in the response would help keep the memory usage lower.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.