Azure / Azure/azure-functions-docker
Durable Functions Memory Issues After Isolated
- Dominant language
- Dockerfile
- Stars
- 280
- Forks
- 126
- PR merge metrics
- No merged PRs in 30d
Description
Hello, I was wondering if someone could help me on some issues I am running into. We have a durable python function app that we have been running via a docker image for a long time now and everything has worked great. Recently after returning to the application we are not able to run it locally anymore. The application will start up and then fail due to memory issues. After some digging I think it may have been [this PR](https://github.com/Azure/azure-functions-docker/pull/1172/files) that caused the issues. Im not certain but its my best guess. I thought it was just a local issue but my entire team is running into the same issue. We are all on M-Series Mac if that plays any role. Below are some of the issues. The application runs fine locally as in func start. This issue is isolated to just when running in the docker container.
Docker Engine v.27.2.0
azure-functions-durable==1.2.9
azure-functions==1.19.0
dockerfile:
```
FROM mcr.microsoft.com/azure-functions/python:4-python3.11
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
AzureFunctionsJobHost__Logging__Console__IsEnabled=true
COPY requirements.txt /
RUN pip install -r /requirements.txt
COPY . /home/site/wwwroot
```
```
info: Host.Startup[412]
Host initialized (207ms)
info: Host.Triggers.DurableTask[0]
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at System.Runtime.CompilerServices.CastHelpers.IsInstanceOfAny(Void*, System.Object)
at System.Collections.Generic.List1[[System.ValueTuple2[[Azure.Core.HttpPipelinePosition, Azure.Core, Version=1.38.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]..ctor(System.Collections.Generic.IEnumerable1>)
at Azure.Core.ClientOptions..ctor(Azure.Core.ClientOptions, Azure.Core.DiagnosticsOptions)
at Azure.Identity.DefaultAzureCredentialOptions..ctor()
at Azure.Identity.DefaultAzureCredentialFactory..ctor(Azure.Identity.DefaultAzureCredentialOptions, Azure.Identity.CredentialPipeline)
at Microsoft.Extensions.Azure.AzureClientsGlobalOptions+<>c.<.ctor>b__11_0(System.IServiceProvider)
at Microsoft.Azure.WebJobs.Script.StorageClientProvider2[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Create(System.String, Microsoft.Extensions.Configuration.IConfiguration, System.__Canon)
at Microsoft.Azure.WebJobs.Script.HostAzureBlobStorageProvider.TryCreateBlobServiceClientFromConnection(System.String, Azure.Storage.Blobs.BlobServiceClient ByRef) at Microsoft.Azure.WebJobs.Script.HostAzureBlobStorageProvider.TryCreateHostingBlobContainerClient(Azure.Storage.Blobs.BlobContainerClient ByRef)
at Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager.CreateBlobContainerClient(System.String)
at Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager.GetContainerClient(System.String)
at Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager+d__9.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager+d__9, Microsoft.Azure.WebJobs.Host.Storage, Version=5.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__9 ByRef)
at Microsoft.Azure.WebJobs.Host.BlobLeaseDistributedLockManager.TryLockAsync(System.String, System.String, System.String, System.String, System.TimeSpan, System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Host.SingletonManager+d__19.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.SingletonManager+d__19, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__19 ByRef)
at Microsoft.Azure.WebJobs.Host.SingletonManager.TryLockAsync(System.String, System.String, Microsoft.Azure.WebJobs.SingletonAttribute, System.Threading.CancellationToken, Boolean)
at Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener+d__13.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener+d__13, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__13 ByRef)
at Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener.StartAsync(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener+d__13.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener+d__13, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__13 ByRef)
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(System.Threading.CancellationToken, Boolean)
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener+d__12.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener+d__12, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__12 ByRef)
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener+d__4.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener+d__4, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__4 ByRef)
at Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener.StartAsync(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener+d__8.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener+d__8, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__8 ByRef)
at Microsoft.Azure.WebJobs.Host.Listeners.ListenerFactoryListener.StartAsyncCore(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener+d__5.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener+d__5, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__5 ByRef)
at Microsoft.Azure.WebJobs.Host.Listeners.ShutdownListener.StartAsync(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.JobHost+d__23.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.JobHost+d__23, Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](d__23 ByRef)
at Microsoft.Azure.WebJobs.JobHost.StartAsyncCore(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Script.ScriptHost+d__97.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Script.ScriptHost+d__97, Microsoft.Azure.WebJobs.Script, Version=4.1036.0.0, Culture=neutral, PublicKeyToken=null]](d__97 ByRef)
at Microsoft.Azure.WebJobs.Script.ScriptHost.StartAsyncCore(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Hosting.JobHostService.StartAsync(System.Threading.CancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host+<b__15_1>d.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Extensions.Hosting.Internal.Host+<b__15_1>d, Microsoft.Extensions.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](<b__15_1>d ByRef)
at Microsoft.Extensions.Hosting.Internal.Host.b__15_1(Microsoft.Extensions.Hosting.IHostedService, System.Threading.CancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host+d__181[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Extensions.Hosting.Internal.Host+d__181[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], Microsoft.Extensions.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](d__181 ByRef)
at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.Collections.Generic.IEnumerable1, System.Threading.CancellationToken, Boolean, Boolean, System.Collections.Generic.List1, System.Func3)
at Microsoft.Extensions.Hosting.Internal.Host+d__15.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Extensions.Hosting.Internal.Host+d__15, Microsoft.Extensions.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](d__15 ByRef)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__58.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__58, Microsoft.Azure.WebJobs.Script.WebHost, Version=4.1036.0.0, Culture=neutral, PublicKeyToken=null]](d__58 ByRef)
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.UnsynchronizedStartHostAsync(Microsoft.Azure.WebJobs.Script.WebHost.ScriptHostStartupOperation, Int32, JobHostStartupMode)
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__57.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__57, Microsoft.Azure.WebJobs.Script.WebHost, Version=4.1036.0.0, Culture=neutral, PublicKeyToken=null]](d__57 ByRef)
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.StartHostAsync(System.Threading.CancellationToken, Int32, JobHostStartupMode, System.Nullable1)
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__54.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService+d__54, Microsoft.Azure.WebJobs.Script.WebHost, Version=4.1036.0.0, Culture=neutral, PublicKeyToken=null]](d__54 ByRef)
at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsScriptHostService.StartAsync(System.Threading.CancellationToken)
at Microsoft.AspNetCore.Hosting.HostedServiceExecutor+d__5.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Hosting.HostedServiceExecutor+d__5, Microsoft.AspNetCore.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](d__5 ByRef)
at Microsoft.AspNetCore.Hosting.HostedServiceExecutor.ExecuteAsync(System.Func2, Boolean)
at Microsoft.AspNetCore.Hosting.WebHost+d__27.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Hosting.WebHost+d__27, Microsoft.AspNetCore.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](d__27 ByRef)
at Microsoft.AspNetCore.Hosting.WebHost.StartAsync(System.Threading.CancellationToken)
at Microsoft.AspNetCore.Hosting.WebHostExtensions+d__5.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Hosting.WebHostExtensions+d__5, Microsoft.AspNetCore.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](d__5 ByRef)
at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(Microsoft.AspNetCore.Hosting.IWebHost, System.Threading.CancellationToken, System.String)
at Microsoft.AspNetCore.Hosting.WebHostExtensions+d__4.MoveNext()
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Hosting.WebHostExtensions+d__4, Microsoft.AspNetCore.Hosting, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60]](d__4 ByRef)
at Microsoft.AspNetCore.Hosting.WebHostExtensions.RunAsync(Microsoft.AspNetCore.Hosting.IWebHost, System.Threading.CancellationToken)
at Microsoft.Azure.WebJobs.Script.WebHost.Program.Main(System.String[])
/opt/startup/start_nonappservice.sh: line 4: 9 Aborted /azure-functions-host/Microsoft.Azure.WebJobs.Script.WebHost
contentapi exited with code 134
```
```
info: Host.Startup[413]
Host started (601ms)
Host.Startup[0]
Job host started
assertion failed [block != nullptr]: BasicBlock requested for unrecognized address
(BuilderBase.h:550 block_for_offset)
/opt/startup/start_nonappservice.sh: line 4: 8 Trace/breakpoint trap /azure-functions-host/Microsoft.Azure.WebJobs.Script.WebHost
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.