googleapis / googleapis/google-cloud-ruby

storage: Error during CRC check when uploading an open + unlinked file

Open
#34,987 0 comments 0 reactions 1 assignee Claimed by @bajajneha27 View on GitHub
api: storage
Dominant language
Ruby
Stars
1.4k
Forks
578
Avg merge
1d 11h
Merged PRs (30d)
166

Description

#### Context

The `google-cloud-storage` library attempts to reopen any `File` that is attempted to be uploaded. See this line:
https://github.com/shubhangi-google/google-cloud-ruby/blob/042468dcccc83a802ae4993e76ebea203827d3cb/google-cloud-storage/lib/google/cloud/storage/file/verifier.rb#L84

**This is a problem when the file no longer exists**.

When would that actually happen? The `puma` web server notably unlinks files immediately after storing, while keeping the file descriptor open:
https://github.com/puma/puma/blob/v8.0.2/lib/puma/client.rb#L507-L511

Although this trips people up, it is very much intentional and in fact recommended by the Ruby documentation.
https://github.com/puma/puma/issues/3597
https://github.com/aws/aws-sdk-ruby/issues/3163
https://github.com/aws/aws-sdk-ruby/pull/3164
https://docs.ruby-lang.org/en/3.4/Tempfile.html#class-Tempfile-label-Unlink+after+creation

It would be tremendously helpful if the `google-cloud-storage` client would not assume that a `File` or `Tempfile` is still linked when it is being uploaded.

#### Environment details

- OS: Alpine Linux, Debian Linux
- Ruby version: 3.4
- Gem name and version: google-cloud-storage (1.62.0)

#### Steps to reproduce

1. Use Puma web server
2. Upload files above configured memory limit
3. Observe that google-cloud-storage fails to compute digest

#### Code example

This is a minimal example that mimics Puma's behaviour but it is a much simpler reproduction.

```ruby
source "https://rubygems.org"

gem "google-cloud-storage", "1.62.0"
```

```ruby
require "google/cloud/storage"
require "tempfile"

storage = Google::Cloud::Storage.anonymous
bucket = storage.bucket("unused", skip_lookup: true)

Tempfile.create("gcs-repro", binmode: true) do |file|
file.write("still readable")
file.rewind

path = file.path
File.unlink(path)

puts "Path exists: #{File.exist?(path)}"
puts "Contents: #{file.read.inspect}"
file.rewind

bucket.create_file(file, "test")
end
```

#### Full backtrace

```sh
3.4-slim: Pulling from library/ruby
b16cb8fde57a: Pulling fs layer
ead741ce7b27: Pulling fs layer
59f54fbcd984: Pulling fs layer
b160435c969d: Pulling fs layer
52a2f7711503: Pulling fs layer
b16cb8fde57a: Download complete
ead741ce7b27: Download complete
b160435c969d: Download complete
4371fa0b2e33: Download complete
79dd7f77afe3: Download complete
52a2f7711503: Download complete
59f54fbcd984: Download complete
59f54fbcd984: Pull complete
ead741ce7b27: Pull complete
b160435c969d: Pull complete
b16cb8fde57a: Pull complete
52a2f7711503: Pull complete
Digest: sha256:614edae6a80eb2a7cf1984f03a6d814f523a48355e8f96deb5ddd25faa86353e
Status: Downloaded newer image for ruby:3.4-slim
/usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'File#initialize': No such file or directory @ rb_sysopen - /tmp/gcs-repro20260721-29-v1priq (Errno::ENOENT)
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'IO.open'
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'Google::Cloud::Storage::File::Verifier._digest_for'
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:59:in 'Google::Cloud::Storage::File::Verifier.crc32c_for'
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/bucket.rb:3507:in 'Google::Cloud::Storage::Bucket#crc32c_for'
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/bucket.rb:2007:in 'Google::Cloud::Storage::Bucket#create_file'
from -e:1:in 'block in '
from /usr/local/lib/ruby/3.4.0/tempfile.rb:576:in 'Tempfile.create_with_filename'
from /usr/local/lib/ruby/3.4.0/tempfile.rb:562:in 'Tempfile.create'
from -e:1:in ''
path exists: false
contents: still readable
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.