Azure / Azure/azure-functions-host

ContainerNotFound during lock lease when running on Kubernetes with KEDA and Azurite

Open
#7,739 1 comment 0 reactions 1 assignee Claimed by @soninaren View on GitHub
Needs: Triage (Functions)
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

When I run my Azure Function with a timer trigger on my local Kubernetes instance (docker-desktop) with KEDA, I get a "ContainerNotFound" error (when requesting blob lock lease on Azurite). When I run the same image on docker-compose everything works.

Here are the logs of Azurite from running on Kubernetes with KEDA:
```
PUT
x-ms-client-request-id:243270b8-c699-4f3b-8954-2290c71c1c89
x-ms-date:Sat, 02 Oct 2021 16:03:27 GMT
x-ms-lease-action:acquire
x-ms-lease-duration:15
x-ms-proposed-lease-id:000000000000000000000000C61506FA
x-ms-return-client-request-id:true
x-ms-version:2020-08-04
/devstoreaccount1/devstoreaccount1/locks/testname5cf5fc8df62gvg9-21373407/host
comp:lease
==> Response Code: 404

---

PUT
application/octet-stream
*
x-ms-blob-type:BlockBlob
x-ms-client-request-id:bf1c5b1e-7a90-4169-a0a1-92fda8861d1e
x-ms-date:Sat, 02 Oct 2021 16:03:27 GMT
x-ms-return-client-request-id:true
x-ms-version:2020-08-04
/devstoreaccount1/devstoreaccount1/locks/testname5cf5fc8df62gvg9-21373407/host
==> Response Code: 404

---

PUT
x-ms-client-request-id:23da69d1-fbc4-438a-8d91-e8a94c8fba59
x-ms-date:Sat, 02 Oct 2021 16:03:27 GMT
x-ms-return-client-request-id:true
x-ms-version:2020-08-04
/devstoreaccount1/devstoreaccount1/azure-webjobs-hosts
restype:container
==> Response Code: 409
```

This are the logs of the same requests with docker-compose without KEDA:
```
PUT
x-ms-client-request-id:6f2e873d-b33b-4eff-879a-0920d2cf0234
x-ms-date:Sat, 02 Oct 2021 17:13:59 GMT
x-ms-lease-action:acquire
x-ms-lease-duration:60
x-ms-proposed-lease-id:d59ff802-5118-471c-bd0c-012d10453c25
x-ms-return-client-request-id:true
x-ms-version:2020-08-04
/devstoreaccount1/devstoreaccount1/azure-webjobs-hosts/locks/4ae6d139a95e-2137340777/***.Poc.Kubernetes.Functions.Functions.Run.Listener
comp:lease
==> Response Code: 404

PUT
application/octet-stream
*
x-ms-blob-type:BlockBlob
x-ms-client-request-id:5ff8856f-7337-4e16-b392-8c1447687218
x-ms-date:Sat, 02 Oct 2021 17:13:59 GMT
x-ms-return-client-request-id:true
x-ms-version:2020-08-04
/devstoreaccount1/devstoreaccount1/azure-webjobs-hosts/locks/4ae6d139a95e-2137340777/***.Poc.Kubernetes.Functions.Functions.Run.Listener
==> Response Code: 404

PUT
x-ms-client-request-id:575d0627-a6fa-4de7-88b7-0ac373a65092
x-ms-date:Sat, 02 Oct 2021 17:13:59 GMT
x-ms-return-client-request-id:true
x-ms-version:2020-08-04
/devstoreaccount1/devstoreaccount1/azure-webjobs-hosts
restype:container
==> Response Code: 201
```

The error is, that on the KEDA setup, locks are requested at `/devstoreaccount1/devstoreaccount1/locks/` but then container creation requests goes at `/devstoreaccount1/devstoreaccount1/azure-webjobs-hosts`. In the docker-compose setup, those are all going at `/devstoreaccount1/devstoreaccount1/azure-webjobs-hosts`.

#### Investigative information

Please provide the following:

- Timestamp: N/A
- Function App version: 3
- Function App name: N/A
- Function name(s) (as appropriate): N/A
- Invocation ID: N/A
- Region: N/A (local)

#### Repro steps

1. Create a Azure Function with a Timer Trigger inside a container
2. Create a docker-compose file, with Azurite in it. Use ist for the `AzureWebJobsStorage` setting. Run it.
3. Create a kubernetes setup (via func cli). Deploy Azurite with a service on your local docker-desktop cluster. `AzureWebJobsStorage` setting. Run it.

#### Expected behavior
Timer Triggered Function runs with docker-desktop setup as well as with Kubernetes KEDA setup.

#### Actual behavior

Timer Triggered Function does not run. Logs:
```
fail: Host.Startup[0]
The listener for function 'TimerTriggeredFunction' was unable to start.
Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException: The listener for function 'TimerTriggeredFunction' was unable to start.
---> Azure.RequestFailedException: The specified container does not exist.
RequestId:275c00e8-52f6-46dc-9e3d-788c98a1b63c
Time:2021-10-03T16:01:22.226Z
Status: 404 (The specified container does not exist.)
ErrorCode: ContainerNotFound

Content:

ContainerNotFound
The specified container does not exist.
RequestId:275c00e8-52f6-46dc-9e3d-788c98a1b63c
Time:2021-10-03T16:01:22.226Z

Headers:
Server: Azurite-Blob/3.14.2
x-ms-error-code: ContainerNotFound
x-ms-request-id: 275c00e8-52f6-46dc-9e3d-788c98a1b63c
Date: Sun, 03 Oct 2021 16:01:22 GMT
Connection: keep-alive
Keep-Alive: REDACTED
Transfer-Encoding: chunked
Content-Type: application/xml

at Azure.Storage.Blobs.BlockBlobRestClient.UploadAsync(Int64 contentLength, Stream body, Nullable`1 timeout, Byte[] transactionalContentMD5, String blobContentType, String blobContentEncoding, String blobContentLanguage, Byte[] blobContentMD5, String blobCacheControl, IDictionary`2 metadata, String leaseId, String blobContentDisposition, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, String encryptionScope, Nullable`1 tier, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, String ifMatch, String ifNoneMatch, String ifTags, String blobTagsString, Nullable`1 immutabilityPolicyExpiry, Nullable`1 immutabilityPolicyMode, Nullable`1 legalHold, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadInternal(Stream content, BlobHttpHeaders blobHttpHeaders, IDictionary`2 metadata, IDictionary`2 tags, BlobRequestConditions conditions, Nullable`1 accessTier, IProgress`1 progressHandler, String operationName, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlockBlobClient.<>c__DisplayClass60_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Azure.Storage.PartitionedUploader`2.UploadInternal(Stream content, TServiceSpecificArgs args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobClient.StagedUploadInternal(Stream content, BlobUploadOptions options, Boolean async, CancellationToken cancellationToken)
at Microsoft.Azure.WebJobs.Script.BlobLeaseDistributedLockManager.TryCreateAsync(BlobLeaseDistributedLockManager lockManager, BlobClient blob, CancellationToken cancellationToken, String accountOverride) in /src/azure-functions-host/src/WebJobs.Script/Host/DistributedLockManagers/BlobLeaseDistributedLockManager.cs:line 250
at Microsoft.Azure.WebJobs.Script.BlobLeaseDistributedLockManager.TryAcquireLeaseAsync(BlobLeaseDistributedLockManager lockManager, BlobClient blobClient, TimeSpan leasePeriod, String proposedLeaseId, CancellationToken cancellationToken, String accountOverride) in /src/azure-functions-host/src/WebJobs.Script/Host/DistributedLockManagers/BlobLeaseDistributedLockManager.cs:line 156
at Microsoft.Azure.WebJobs.Script.BlobLeaseDistributedLockManager.TryLockAsync(String account, String lockId, String lockOwnerId, String proposedLeaseId, TimeSpan lockPeriod, CancellationToken cancellationToken) in /src/azure-functions-host/src/WebJobs.Script/Host/DistributedLockManagers/BlobLeaseDistributedLockManager.cs:line 82
at Microsoft.Azure.WebJobs.Host.SingletonManager.TryLockAsync(String lockId, String functionInstanceId, SingletonAttribute attribute, CancellationToken cancellationToken, Boolean retry) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonManager.cs:line 113
at Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener.StartAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Singleton\SingletonListener.cs:line 48
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(CancellationToken cancellationToken, Boolean allowRetry) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 68
--- End of inner exception stack trace ---
```

#### Known workarounds
Use an Azure Blob Storage Account Connection String for `AzureWebJobsStorage` which works on both setups.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.