Azure / Azure/azure-functions-host
[V1] Function App can get in bad state if unable to create directory for FileSystemSecretsRepo
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
The default for V1 function apps is to store secrets in a FileSecretsRepo. This issue is related to a CRI where the customer's app was put in a bad state and unable to recover because it was unable to create a directory for these secrets. The exceptions were coming from the FileSystemSecretsRepo constructor [here](https://github.com/Azure/azure-functions-host/blob/v1.x/src/WebJobs.Script.WebHost/Security/FileSystemSecretsRepository.cs#L38):
```
System.IO.IOException : The network path was not found.
at System.IO.__Error.WinIOError(Int32 errorCode,String maybeFullPath)
at System.IO.Directory.InternalCreateDirectory(String fullPath,String path,Object dirSecurityObj,Boolean checkHost)
at System.IO.Directory.InternalCreateDirectoryHelper(String path,Boolean checkHost)
at System.IO.Directory.CreateDirectory(String path)
at Microsoft.Azure.WebJobs.Script.WebHost.FileSystemSecretsRepository..ctor(String secretsPath) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\FileSystemSecretsRepository.cs : 38
```
```
Autofac.Core.DependencyResolutionException : An error occurred during the activation of a particular registration. See the inner exception for details. Registration: Activator = WebScriptHostManager (DelegateActivator), Services = [Microsoft.Azure.WebJobs.Script.WebHost.WebScriptHostManager], Lifetime = Autofac.Core.Lifetime.CurrentScopeLifetime, Sharing = None, Ownership = ExternallyOwned ---> Host failed to initialize. (See inner exception for details.) ---> Microsoft.Azure.WebJobs.Script.HostInitializationException : Host failed to initialize. ---> System.IO.IOException : The network path was not found.
at System.IO.__Error.WinIOError(Int32 errorCode,String maybeFullPath)
at System.IO.Directory.InternalCreateDirectory(String fullPath,String path,Object dirSecurityObj,Boolean checkHost)
at System.IO.Directory.InternalCreateDirectoryHelper(String path,Boolean checkHost)
at System.IO.Directory.CreateDirectory(String path)
at Microsoft.Azure.WebJobs.Script.WebHost.FileSystemSecretsRepository..ctor(String secretsPath) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\FileSystemSecretsRepository.cs : 38
```
The fix here would involve making this code path more resilient. There are times, depending on the hosting plan, where the file system could become read only for a period of time so retry logic could help. At the very least, we should make sure the host doesn't get stuck in this unrecoverable state.
Some potential improvements --
- Adding retry logic in case it's a transient issue
- Make sure the host recycles if it's unable to recover
Contributor guide
Research direction
Start with src/WebJobs.Script.WebHost/Security/FileSystemSecretsRepository.cs, especially the constructor at line 38, and trace how its failure prevents host initialization. Review the V1 host initialization path and the reported IOException behavior. Done means the host can recover from a temporary or unavailable secrets directory rather than remaining in an unrecoverable state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100