envoyproxy / envoyproxy/envoy

Filesystem xDS on Windows triggers permission errors in the updater

Open
#30,378 5 comments 0 reactions 0 assignees View on GitHub
bug no stalebot
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

*Description*:

I have observed inconsistent but reproducible errors while updating xDS files on Windows (using Python's `os.replace()`, but I expect any atomic overwrite would hit a similar error):

```
Traceback (most recent call last):
.......
File "envoy_config.py", line 230, in replace_config_file
os.replace(source, dest)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Dev\\AppData\\Local\\Temp\\envoy-config-258r5e6p\\tmpzdckoqi_' -> 'C:\\Users\\Dev\\AppData\\Local\\Temp\\envoy-config-258r5e6p\\tmpgkg8r9ni-cds.yaml'
```

If I am not mistaken this error happens when the xDS file is updated while Envoy is still reading the file from a previous update, during which time Envoy effectively holds a lock on the file. I _believe_ this could be [avoided](https://stackoverflow.com/q/5891053/113632) by configuring [file sharing](https://learn.microsoft.com/en-us/dotnet/api/system.io.fileshare), but I have not tested this.

Although this error arises in the separate updater process I feel this is an Envoy bug because there is no reasonable way for the updater process to know when it is safe to initiate an update. Workarounds involve retrying and/or sleeping, neither of which are ideal.

Notably, nothing in the Envoy documentation mentions this limitation, so if this _is_ considered WAI the documentation should be updated to clarify how well-behaved updaters should be implemented.

*Repro steps*:
1. Run a filesystem-backed xDS config on Windows
2. In a tight loop update the xDS files repeatedly using Python's `os.replace()` or similar
3. Observe permission errors in the process updating the xDS files

Here is an example triggering this error without Envoy, just to demonstrate that holding a file reference open is sufficient:

```
PS> python3.8.exe
Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import tempfile
>>> dest=tempfile.NamedTemporaryFile(suffix="-dest", delete=False).name
>>> os.replace(tempfile.NamedTemporaryFile(suffix="-tmp", delete=False).name, dest)
>>> f = open(dest)
>>> os.replace(tempfile.NamedTemporaryFile(suffix="-tmp", delete=False).name, dest)
Traceback (most recent call last):
File "", line 1, in
PermissionError: [WinError 5] Access is denied: 'tmpuau8nzjn-tmp' -> 'tmpq3lqearn-dest'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.