Azure / Azure/azure-rest-api-specs

BUG: Create PrivateEndpoint using PrivateLink's alias may fail in concurrent creation scenario

Open
#20,289 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Network - Private Link Service Attention
Dominant language
TypeSpec
Stars
3.1k
Forks
6k
Avg merge
2d 22h
Merged PRs (30d)
444

Description

When creating multiple PrivateEndpoints with PrivateLink's alias may fail with the following error message:

```text
PrivateLinkServiceId Invalid private link service id: {0} passed in private link service connection: {1} passed in input is invalid
```

The put request to create the PrivateLink resource:

```text
PUT https://management.azure.com/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateLinkServices/acctestPLS-220818165005420368?api-version=2021-08-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v65.0.0 network/2021-08-01 HashiCorp Terraform/1.2.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 962
Authorization: Bearer
Content-Type: application/json; charset=utf-8
X-Ms-Authorization-Auxiliary:
X-Ms-Correlation-Request-Id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
Accept-Encoding: gzip
```

The request's body is:

```json
{
"location": "centralus",
"properties": {
"autoApproval": {
"subscriptions": [
"xxxxxxxxxxxxx"
]
},
"enableProxyProtocol": false,
"fqdns": [],
"ipConfigurations": [
{
"name": "primaryIpConfiguration-220818165005420368",
"properties": {
"primary": true,
"privateIPAddress": "",
"privateIPAddressVersion": "IPv4",
"privateIPAllocationMethod": "Dynamic",
"subnet": {
"id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/virtualNetworks/acctestvnet-220818165005420368/subnets/acctestsnetservice-220818165005420368"
}
}
}
],
"loadBalancerFrontendIpConfigurations": [
{
"id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/loadBalancers/acctestlb-220818165005420368/frontendIPConfigurations/acctestpip-220818165005420368"
}
],
"visibility": {
"subscriptions": [
"xxxxxxxxxxxxx"
]
}
},
"tags": {}
}
```

The response is:

```text
HTTP/1.1 201 Created
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 2585
Content-Type: application/json; charset=utf-8
Expires: -1
Retry-After: 10
x-ms-request-id: 2da20903-2d40-4b7b-b545-50dfb9602eab
Azure-AsyncOperation: https://management.azure.com/subscriptions/xxxxxxxxxxxxx/providers/Microsoft.Network/locations/centralus/operations/2da20903-2d40-4b7b-b545-50dfb9602eab?api-version=2021-08-01
x-ms-correlation-request-id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
Azure-AsyncNotification: Enabled
x-ms-arm-service-request-id: 4bc3f756-b4da-4b91-95b9-522032985f74
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-writes: 1195
x-ms-routing-request-id: SOUTHEASTASIA:20220818T085145Z:33bc72f4-61e9-47bc-854d-ec7d23274951
X-Content-Type-Options: nosniff
Date: Thu, 18 Aug 2022 08:51:44 GMT

{
"name": "acctestPLS-220818165005420368",
"id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateLinkServices/acctestPLS-220818165005420368",
"etag": "W/\"067e8de9-b44f-473a-901d-0ad58301f3cc\"",
"type": "Microsoft.Network/privateLinkServices",
"location": "centralus",
"tags": {},
"properties": {
"provisioningState": "Updating",
"resourceGuid": "e2736d14-8e82-4051-98d0-cbff65b71fae",
"fqdns": [],
"alias": "acctestpls-220818165005420368.b08a765d-0bcb-41bf-a0ec-755d43737dbc.centralus.azure.privatelinkservice",
"visibility": {
"subscriptions": [
"xxxxxxxxxxxxx"
]
},
"autoApproval": {
"subscriptions": [
"xxxxxxxxxxxxx"
]
},
"enableProxyProtocol": false,
"loadBalancerFrontendIpConfigurations": [
{
"id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/loadBalancers/acctestlb-220818165005420368/frontendIPConfigurations/acctestpip-220818165005420368"
}
],
"ipConfigurations": [
{
"name": "primaryIpConfiguration-220818165005420368",
"id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateLinkServices/acctestPLS-220818165005420368/ipConfigurations/primaryIpConfiguration-220818165005420368",
"etag": "W/\"067e8de9-b44f-473a-901d-0ad58301f3cc\"",
"type": "Microsoft.Network/privateLinkServices/ipConfigurations",
"properties": {
"provisioningState": "Succeeded",
"privateIPAllocationMethod": "Dynamic",
"subnet": {
"id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/virtualNetworks/acctestvnet-220818165005420368/subnets/acctestsnetservice-220818165005420368"
},
"primary": true,
"privateIPAddressVersion": "IPv4"
}
}
],
"privateEndpointConnections": [],
"networkInterfaces": [
{
"id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/networkInterfaces/acctestPLS-220818165005420368.nic.fd797a3e-c0fd-4464-a8cc-32a3f1e21992"
}
]
}
}
```

As we can see, the PrivateLink resource `acctestPLS-220818165005420368` was created successfully, it's alias was `acctestpls-220818165005420368.b08a765d-0bcb-41bf-a0ec-755d43737dbc.centralus.azure.privatelinkservice`

Then we sent a request to create a new PrivateEndpoint using this alias:

```text
PUT https://management.azure.com/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/privateEndpoints/acctest-privatelink-220818165005420368?api-version=2021-08-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v65.0.0 network/2021-08-01 HashiCorp Terraform/1.2.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 593
Authorization: Bearer
Content-Type: application/json; charset=utf-8
X-Ms-Authorization-Auxiliary:
X-Ms-Correlation-Request-Id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
Accept-Encoding: gzip
```

The body was:

```json
{
"location": "centralus",
"properties": {
"manualPrivateLinkServiceConnections": [
{
"name": "acctestPLS-220818165005420368",
"properties": {
"groupIds": [],
"privateLinkServiceId": "acctestpls-220818165005420368.b08a765d-0bcb-41bf-a0ec-755d43737dbc.centralus.azure.privatelinkservice",
"requestMessage": "test"
}
}
],
"privateLinkServiceConnections": [],
"subnet": {
"id": "/subscriptions/xxxxxxxxxxxxx/resourceGroups/acctestRG-privatelink-220818165005420368/providers/Microsoft.Network/virtualNetworks/acctestvnet-220818165005420368/subnets/acctestsnetendpoint-220818165005420368"
}
},
"tags": {}
}
```

We used the PrivateLink's alias here, and the response was:

```text
HTTP/1.1 400 Bad Request
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 237
Content-Type: application/json; charset=utf-8
Expires: -1
x-ms-request-id: d8c3b7fa-7169-4ce5-b2ae-4ecc7629d94c
x-ms-correlation-request-id: 0c5863fa-fb25-5b75-eab0-4802ca30531e
x-ms-arm-service-request-id: 3edd9ac3-4ceb-42b1-b1d2-92ca978897df
Strict-Transport-Security: max-age=31536000; includeSubDomains
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-writes: 1196
x-ms-routing-request-id: SOUTHEASTASIA: 20220818T085206Z:c744abc8-4534-4774-845a-a588a6856e90
X-Content-Type-Options: nosniff
Date: Thu,
18 Aug 2022 08: 52: 06 GMT

{
"error": {
"code": "InvalidPrivateLinkServiceId",
"message": "PrivateLinkServiceId Invalid private link service id: {0} passed in private link service connection: {1} passed in input is invalid",
"details": []
}
}
```

We believe this issue is caused by some cache-related issue on service side because:

1. It only happens when we create multiple PrivateLinks concurrently
2. If we retry the creation of PrivateEndpoint, we'll get a successful creation eventually.

We can reproduce this issue by creationg multiple PrivateLinks concurrently, then create a PrivateEndpoint with PrivateLink's alias. Can anyone help us solve this issue? Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository file or test is identified. Start by reproducing the concurrent PrivateLink creation and subsequent PrivateEndpoint PUT using the request bodies and alias shown here, then compare retry behavior. Done would require determining a concrete repository change or documenting that the failure is entirely service-side; the issue does not define either outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
api, cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.