googleapis / googleapis/google-cloud-java
[java-storage] It's impossible to reliably upload a complete file
- 主要语言
- Java
- 星标
- 2.1k
- 派生
- 1.2k
- 平均合并
- 1 天 23 小时
- 30 天内合并 PR
- 157
描述
We're trying to upload a file to GCS, and make sure the file is only published if the upload succeeds. Consider this code:
```java
BlobInfo blobInfo = ...;
ByteBuffer buf1, buf2;
WriteChannel writer = storage.writer(blobInfo, Storage.BlobWriteOption.crc32cMatch());
try {
writer.write(buf1);
writer.write(buf2); // network goes down before this call
} finally {
writer.close(); // network is back up before this call
}
```
The write of `buf1` is successful, but before writing `buf2`, the network goes down. As expected, an `IOException` is thrown (both my buffers are 32MB). The execution proceeds to the `finally` clause to call `close()`. At this point, the network is up again. The `close` call finishes the file upload, and the file is published with partial contents!
However, the problem is more generic. While writing the file, any error can occur (e.g. an IO error while reading the file to upload). In this case, the `close()` call will also publish a partial file.
I have tested moving the `close()` call to the `try` block, so that the `WriteChannel` is abandoned without closing it. This seems to have the desired effect, but since the `close()` method is inherited from `AutoCloseable`, it's very unintuitive, and some code quality tools will suggest or force calling the `close()` method.
One possible backward-compatible fix to this problem would be to add `abort()` method, that will clean up the resources without finishing the upload, and after which `close()` will be no-op. That method could be automatically called for internal exceptions.
I'm using version 2.50.0 on Java 21.
贡献指南
调研方向
从 Java 存储入口 `Storage.writer` 以及报告中描述的 `WriteChannel.close()` 行为开始。重现 `ByteBuffer` 上传失败场景,并确定写入错误后清理应如何进行;当失败的上传无法发布部分内容,同时资源仍得到清理时,即视为完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- google-cloud, java
- 领域
- cloud
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100