googleapis / googleapis/google-api-java-client

Improve memory efficiency of base64-encoded byte array decoding operations

Đang mở
#967 4 bình luận 0 reaction 0 người được giao Xem trên GitHub
priority: p4 type: feature request
Ngôn ngữ chính
Java
Star
1.5k
Fork
772
Merge trung bình
1 giờ 44 phút
Pull request đã merge (30 ngày)
1

Mô tả

Currently, decoding base64-encoded byte array with the Java API client is very inefficient in terms of memory usage.

Reading big attachments from the GMail API in memory-constrained environments like App Engine is quite a challenge because of that.
https://developers.google.com/gmail/api/v1/reference/users/messages/attachments/get

On a 64 bit Java 7 VM, it required at least 115 MB of heap (-Xmx115M) to read a 12Mb attachment from the Gmail API (which is a 17MB base64 string in the response from the API).

The code to test it is dead simple (MESSAGE_ID / ATTACHMENT_ID references a big attachment):

``` java
gmail.users().messages().attachments().get("me", MESSAGE_ID, ATTACHMENT_ID).execute()
```

Here is the stacktrace when trying to load the attachment with only 110MB or heap:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:2694)
at java.lang.String.(String.java:203)
at java.lang.StringBuilder.toString(StringBuilder.java:405)
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:360)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:277)
at com.google.api.client.json.jackson2.JacksonParser.getText(JacksonParser.java:76)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:850)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:471)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:780)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:381)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:354)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:87)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:81)
at com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:459)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at LoadEmail.main(LoadEmail.java:31)

Obtaining a stream from the data field in the response would help keep the memory usage lower.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.