GoogleCloudPlatform / GoogleCloudPlatform/gcsfuse
Any way to reduce duplicate CreateObject calls on file creation?
- Dominant language
- Go
- Stars
- 2.3k
- Forks
- 510
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 35
Description
Hi! I notice that when creating a new file in a mounted directory, there are two CreateObject calls made.
The first call is from the fuse operation "CreateFile", and goes to CreateChildFile in internal/fs/inode/dir.go#851 which creates an empty backing object.
The second call is from the fuse operation "FlushFile", which goes to internal/gcsx/syncer.go to actually create the file with its contents in place.
Our application can sometimes be write-heavy, and during writes, the total time is dominated by the CreateObject calls, because they happen sequentially.
Is there a way to make it so that writes only do one CreateObject request instead of 2, perhaps by eliminating remote creation of the empty backing object? I understand it was perhaps done to increase consistency, but this seems like it would decrease the latency on file writes by 40-50%.
Here's some logs showing the behavior:
```fuse_debug: 2021/10/05 15:47:47.907113 Op 0x0000000b connection.go:411] <- GetInodeAttributes (inode 1, PID 14944)
fuse_debug: 2021/10/05 15:47:47.907359 Op 0x0000000b connection.go:493] -> OK ()
fuse_debug: 2021/10/05 15:47:47.907464 Op 0x0000000c connection.go:411] <- LookUpInode (parent 1, name "test.txt", PID 14944)
gcs: 2021/10/05 15:47:47.907738 Req 0x7: <- StatObject("test.txt/")
gcs: 2021/10/05 15:47:47.907827 Req 0x8: <- StatObject("test.txt")
2021/10/05 15:47:47.931042 Not retrying StatObject("test.txt/") after error of type *gcs.NotFoundError ("gcs.NotFoundError: googleapi: Error 404: No such object: weirdglooptest
/test.txt/, notFound"): &gcs.NotFoundError{Err:(*googleapi.Error)(0xc00052a300)}
gcs: 2021/10/05 15:47:47.931077 Req 0x7: -> StatObject("test.txt/") (23.363965ms): gcs.NotFoundError: googleapi: Error 404: No such object: weirdglooptest/test.txt
/, notFound
2021/10/05 15:47:47.931453 Not retrying StatObject("test.txt") after error of type *gcs.NotFoundError ("gcs.NotFoundError: googleapi: Error 404: No such object: weirdglooptest/
test.txt, notFound"): &gcs.NotFoundError{Err:(*googleapi.Error)(0xc00052a360)}
gcs: 2021/10/05 15:47:47.931474 Req 0x8: -> StatObject("test.txt") (23.642335ms): gcs.NotFoundError: googleapi: Error 404: No such object: weirdglooptest/test.txt,
notFound
fuse_debug: 2021/10/05 15:47:47.931552 Op 0x0000000c connection.go:495] -> Error: "no such file or directory"
fuse_debug: 2021/10/05 15:47:47.931636 Op 0x0000000d connection.go:411] <- CreateFile (parent 1, name "test.txt", PID 14944)
gcs: 2021/10/05 15:47:47.931844 Req 0x9: <- CreateObject("test.txt")
gcs: 2021/10/05 15:47:48.078001 Req 0x9: -> CreateObject("test.txt") (146.149426ms): OK
fuse_debug: 2021/10/05 15:47:48.078074 Op 0x0000000d connection.go:493] -> OK (inode 3)
fuse_debug: 2021/10/05 15:47:48.078211 Op 0x0000000e connection.go:411] <- FlushFile (inode 3, PID 14944)
fuse_debug: 2021/10/05 15:47:48.079100 Op 0x0000000e connection.go:493] -> OK ()
fuse_debug: 2021/10/05 15:47:48.079965 Op 0x0000000f connection.go:411] <- WriteFile (inode 3, PID 0, handle 1, offset 0, 2 bytes)
fuse_debug: 2021/10/05 15:47:48.080069 Op 0x00000010 connection.go:411] <- SetInodeAttributes (inode 3, PID 14944, mtime 2021-10-05 15:47:48.076569479 +0000 UTC)
gcs: 2021/10/05 15:47:48.080653 Req 0xa: <- Read("test.txt", )
gcs: 2021/10/05 15:47:48.104184 Req 0xa: -> Read("test.txt", ) (23.541026ms): OK
fuse_debug: 2021/10/05 15:47:48.104288 Op 0x0000000f connection.go:493] -> OK ()
fuse_debug: 2021/10/05 15:47:48.104411 Op 0x00000010 connection.go:493] -> OK ()
fuse_debug: 2021/10/05 15:47:48.104926 Op 0x00000011 connection.go:411] <- FlushFile (inode 3, PID 14944)
gcs: 2021/10/05 15:47:48.105159 Req 0xb: <- CreateObject("test.txt")
gcs: 2021/10/05 15:47:48.292903 Req 0xb: -> CreateObject("test.txt") (187.740338ms): OK
fuse_debug: 2021/10/05 15:47:48.293031 Op 0x00000011 connection.go:493] -> OK ()
fuse_debug: 2021/10/05 15:47:48.293322 Op 0x00000012 connection.go:411] <- ReleaseFileHandle (PID 0)
fuse_debug: 2021/10/05 15:47:48.293513 Op 0x00000012 connection.go:493] -> OK ()```
Contributor guide
Assessment
This issue has not been assessed yet.