apache / apache/arrow-rs-object-store
On Windows, rapidly trying to create the same file results in "Access is denied" errors
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Describe the bug**
Rapidly creating the same file (as seen in the reproduction section) results in this error:
```
Generic LocalFileSystem error: Unable to open file test_dir\test\test.bin#2: Access is denied. (os error 5)
Generic LocalFileSystem error: Unable to open file test_dir\test\test.bin#1: Access is denied. (os error 5)
Generic LocalFileSystem error: Unable to open file test_dir\test\test.bin#3: Access is denied. (os error 5)
Generic LocalFileSystem error: Unable to open file test_dir\test\test.bin#5: Access is denied. (os error 5)
Generic LocalFileSystem error: Unable to open file test_dir\test\test.bin#2: Access is denied. (os error 5)
[...]
```
**To Reproduce**
```rust
use object_store::local::LocalFileSystem;
use object_store::path::Path;
use object_store::{Error, ObjectStore, PutMode, PutPayload};
#[tokio::main]
async fn main() {
let store = LocalFileSystem::new_with_prefix("./test_dir").unwrap();
for _ in 0..16 {
let store = store.clone();
tokio::spawn(async move {
loop {
match store
.put_opts(
&Path::parse("test/test.bin").unwrap(),
PutPayload::new(),
PutMode::Create.into(),
)
.await
{
Ok(_) | Err(Error::AlreadyExists { .. }) => (),
Err(err) => println!("{err}"),
}
}
});
}
tokio::time::sleep(std::time::Duration::from_secs_f32(1.5)).await;
}
```
with
```toml
[package]
name = "object-store-test"
version = "0.1.0"
edition = "2024"
[dependencies]
object_store = "0.13.2"
tokio = { version = "1.43.0", features = ["bytes", "default", "fs", "io-std", "io-util", "libc", "macros", "mio", "net", "rt", "rt-multi-thread", "socket2", "sync", "time", "tokio-macros"] }
```
and creating the `test_dir` directory, then `cargo run`.
The `tokio` features can likely be reduced by a lot, this is just what I copied from the `Cargo.toml` of the bigger project this occurred in, but I also doubt they are related to the error.
**Expected behavior**
The only expected error would've been `Error::AlreadyExists`, but it seemingly often fails before that when creating the staging file (so this is probably unrelated to the `PutMode` as well and would show up when trying to recreate the file rapidly, though that was not tested).
**Additional context**
I'm not sure if this may "really" be a bug in Rusts `std` or maybe even Windows & not `object_store`; based on a little digging I assume the error comes from `CreateFileW`, which does mention this:
> If you call CreateFile on a file that is pending deletion as a result of a previous call to [DeleteFile](https://learn.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-deletefilea), the function fails. The operating system delays file deletion until all handles to the file are closed. [GetLastError](https://learn.microsoft.com/en-us/windows/desktop/api/errhandlingapi/nf-errhandlingapi-getlasterror) returns ERROR_ACCESS_DENIED.
I'm not sure if trying to rename a file (to the final path) actually counts as a deletion in Windows though, but at least on https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew I didn't find any more info that seems relevant to this exact issue, and I didn't search further than that either.
Contributor guide
Research direction
Start with the LocalFileSystem::put_opts entry point and run the supplied concurrent reproduction on Windows. Trace the staging-file creation and final rename behavior, then verify that repeated creates produce Error::AlreadyExists rather than Windows access-denied errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100