Filesystem xDS on Windows triggers permission errors in the updater
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 430
Description
*Description*:
I have observed inconsistent but reproducible errors while updating xDS files on Windows (using Python's `os.replace()`, but I expect any atomic overwrite would hit a similar error):
```
Traceback (most recent call last):
.......
File "envoy_config.py", line 230, in replace_config_file
os.replace(source, dest)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Dev\\AppData\\Local\\Temp\\envoy-config-258r5e6p\\tmpzdckoqi_' -> 'C:\\Users\\Dev\\AppData\\Local\\Temp\\envoy-config-258r5e6p\\tmpgkg8r9ni-cds.yaml'
```
If I am not mistaken this error happens when the xDS file is updated while Envoy is still reading the file from a previous update, during which time Envoy effectively holds a lock on the file. I _believe_ this could be [avoided](https://stackoverflow.com/q/5891053/113632) by configuring [file sharing](https://learn.microsoft.com/en-us/dotnet/api/system.io.fileshare), but I have not tested this.
Although this error arises in the separate updater process I feel this is an Envoy bug because there is no reasonable way for the updater process to know when it is safe to initiate an update. Workarounds involve retrying and/or sleeping, neither of which are ideal.
Notably, nothing in the Envoy documentation mentions this limitation, so if this _is_ considered WAI the documentation should be updated to clarify how well-behaved updaters should be implemented.
*Repro steps*:
1. Run a filesystem-backed xDS config on Windows
2. In a tight loop update the xDS files repeatedly using Python's `os.replace()` or similar
3. Observe permission errors in the process updating the xDS files
Here is an example triggering this error without Envoy, just to demonstrate that holding a file reference open is sufficient:
```
PS> python3.8.exe
Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import tempfile
>>> dest=tempfile.NamedTemporaryFile(suffix="-dest", delete=False).name
>>> os.replace(tempfile.NamedTemporaryFile(suffix="-tmp", delete=False).name, dest)
>>> f = open(dest)
>>> os.replace(tempfile.NamedTemporaryFile(suffix="-tmp", delete=False).name, dest)
Traceback (most recent call last):
File "", line 1, in
PermissionError: [WinError 5] Access is denied: 'tmpuau8nzjn-tmp' -> 'tmpq3lqearn-dest'
```
Contributor guide
Assessment
This issue has not been assessed yet.