Proposal - Bicep Extensibility (Phase 1)

# Proposal - Bicep Extensibility (Phase 1)
This document outlines the changes that are needed to Bicep, ARM Template JSON, and the ARM Deployment engine in order to support the ability to deploy resources outside of the ARM control plane. The Bicep syntax is liable to change, and will go through the normal design process.

**Phase 1** relates to enabling a limited number of built-in 1st party providers. **Phase 2** will permit the enablement of 3rd party providers.

## Purpose / Goals
* Permit authoring and deploying of custom (non-ARM) resource types through Bicep as part of an ARM Template deployment, with a feature set as full as the features available for ARM resources.
* Leave the door open for 3rd party extensibility with a design which is compatible.

## Non-goals
* Fleshing out the details for 3rd party extensibility.
* Server-side hosting of multiple versions of the same provider code.
* Coming up with a packaging/distribution model for provider code. It is assumed that we will same package management framework as for the BMR effort, when it is ready.
* Support for a 'long-running operation' equivalent.

## Bicep Syntax
### Importing / Configuring extensions
First party extensions will be directly embedded in Bicep, but not enabled in a particular file without an `import` statement. Unless a provider is explicity imported in a file, it will be disabled, and its types will not be available for completions, validation etc.

#### Import syntax
The import statement will allow optional aliasing of an extension (with the `as` statement), and optional configuration (with an object body following the import). Configuration may be necessary depending on the provider to configure connection details - e.g. hostnames, ip addressses, connection strings.

Where possible, configuration should be minimal, especially for first-party services where OBO may be used for authentication.

For example:
```bicep
// example with aliasing + configuration
import kubernetes as k8s {
hostname: ...
}

// example with no aliasing or configuration
import aad
```

We will require the provider configuration block to be a deploy-time constant - e.g. sourced from constants or parameters.

> **NOTE**: We should plan what the versioned syntax (for phase 2) will look like, to make sure this syntax is future-proof. Discuss whether we should enforce this even in phase 1.

> **NOTE**: Copy+pasting the same `import` statement may be quite verbose when working with multiple files; we may want to consider allowing an imported provider to be passed as a parameter.

### Type strings
Extensible resource types will be referenced in Bicep with a unique 'type string'. The 'provider' portion is used to uniquely identify an extensibility provider, and the 'type' & 'version' portions are used to uniquely identify resource types exposed by that provider.

The type string will take the following format:
```
{provider}:{type}@{version}
```

Example:
```
aad:application@1.0
```

### Deploying and referencing
The bicep resource syntax is unchanged, though the structure of the resource body will be defined by the provider types, and will not necessarily follow the traditional `name`/`properties` format used for ARM resources.

```bicep
resource aadApp 'aad:application@1.0' = {
uniqueName: 'myAadApp'
...
}

resource aadSp 'aad:servicePrincipal@1.0' = {
appId: aadApp.appId
...
}
```

### Referencing an existing resource
```bicep
resource aadApp 'aad:application@1.0' existing = {
uniqueName: 'myAadApp'
}
```

## Implementation - Extensibility Provider Contract
### Client-side (Bicep CLI, IDE)
Client-side type validation will be provided by a static JSON file, using the format defined in https://github.com/Azure/bicep-types-az/tree/main/src/Bicep.Types.

This provides functionality equivalent to the following methods:
* `ListTypes()` - returns a list of all of the available `type_strings` provided by the plugin.
* `GetType(type_string)` - fetches the type definition for a given `type_string`.
* `GetConfigType()` - returns the type definition for the provider configuration block.

An additional `ObjectPropertyFlags` enum value will be added to annotate an object property which composes an identifier, in order to declare property(s) which are necessary for referencing an `existing` resource.

The benefits of this approach:
* Language-agnostic: we avoid introducing a requirement for 3rd party extensions to be authorbed in C#.
* Exhaustively testable: we can iterate through the full set of types and ensure we won't encounter unexpected exceptions.
* Potentially safer: we don't have to worry about executing arbitrary code client-side.
* Simpler type generation: programmatic generation from API definitions is simpler.

### Server-side
The server-side extensibility code will be implemented in C#, with a contract that would be easily translated to a wire format (JSONRPC) so that future implementations need not be implemented in C#, and can be executed remotely, or e.g. in a container.

The following methods **MUST** be defined for each resource type:
* `Save(ResourceData)` - performs an idempotent upsert on a resource.
* `PreviewSave(ResourceData)` - previews `Save()` for a resource - combining preflight & what-if functionality without actually executing the operation.
* `Get(ResourceData)` - performs a get on a resource.
* `Delete(ResourceData)` - performs a resource deletion. Not needed for the current spec, but may be necessary in future - e.g. for compatibility with Stacks.

Extensibility provider DLLs **MUST** be embedded in the Deployment engine codebase and at a version equal or greater than the version being requested in a deployment.

It is also valid for the provider to define resoures which **only** define `Get()`, for the purposes of retrieving information from an API which is inherently read-only.

#### Failure Handling
As part of the error contract between Deployments & extensibility provider, the provider should be able to indicate whether or not a failure is transient, along with a retry period. ARM should not generically retry any failure without this indication, and an extensibility provider is not expected to handle transient failures internally. This should give a consistent experience with visibility into failures between extensions, and avoid potential pitfalls where both ARM & provider implement retries.

#### ResourceData type definition
Type definition for the `ResourceData` definition described in this contract:
```ts
type ResourceData = {
// extensibility provider information
import: {
// the provider name
provider: string,
// the provider version
version: string,
// provider-specific configuration data
config: object,
},
// the resource type string
type: string,
// the resource body data
properties: object,
};
```

#### JSONRPC translation
The above methods can be represented as JSONRPC operations with `resource/` - e.g. `resource/save`, `resource/previewSave` etc. Example JSONRPC operation:
```json
{
"jsonrpc": "2.0",
"id": 1,
"method": "resource/previewSave",
"params": {
"type": "application@1.0",
"import": {
"provider": "AAD",
"version": "0.1",
"config": {
...
},
},
"properties": {
...
}
}
}
```

#### Implementation Details
* Until provider versioning is supported server-side (phase 2), provider changes must follow similar guidelines to ARM RP versioning regarding breaking-changes and backwards compatibility within a supported resource type version.
* It is the job of the provider implementation to abstract away any complexities around idempotency behind the `Save()` operation, and only return a response when the idempotent operation is considered completed, or failed. Examples of such complexity that should not be exposed include insert-or-update logic, long-running operation polling, resumption of a retried operation.
* Certain deployment-instruction properties may be generically added to type definitions and not exposed to the provider - currently limited to the `dependsOn` property.

> **NOTE**: Bicep injects pseudo-properties into type definitions (for example, `dependsOn`). We may need to consider a syntax to disambiguate between "Bicep-injected" vs provider-authored to account for naming clashes.

#### Namespace Methods
This specification does not provide any mechanism for an extensibility provider to expose custom methods (e.g. the equivalent to `resourceId()` or `reference()` under the `az` namespace). This precludes the ability to move the `az` namespace functionality into a 1st party extension.

> **NOTE**: Have a discussion on whether this is a requirement for phase 1.

## Template JSON mechanics
Since Bicep is compiled to ARM JSON, a representation is needed for resource declarations and resource referencing.

> **NOTE**: This takes a dependency on the template engine's symbolic name support.

### Declaring a resource

* `type`: The resource `type_string`.
* `import`: Provides information about the provider plugin.
* `provider`: The unique extensibility provider name.
* `version`: The version of the named extensibility provider.
* `config`: Provider-specific configuration.
* `properties`: The resource definition body. This will generally be a 1:1 with the Bicep definition, but potentially containing ARM expressions or control flow statements (e.g. `copy` loops).

```json
{
"type": "application@1.0",
"import": {
"provider": "AAD",
"version": "0.1",
"config": {
...
}
},
"properties": {
...
}
}
```

> **NOTE**: `config` will contain a lot of repetition if there are many resources deployed for the same provider. Bicep should aim to generate variable blocks to avoid repetition, but this does introduce complexity around handling naming conflicts.
> **NOTE**: The `config` section is likely to contain secure information. Do we need to provide a mechanism to securely hide this information, or is it enough to assume that sensitive values will come from secure parameters or run-time values (e.g. `listKeys`)?

### Referencing a resource in-template
In order to reference an extensible resource in-template, we will leverage symbolic name support, where `resName` is the symbolic name of the resource, rather than obtained from properties in the resource body:

```
[reference('resName')]
```

### Referencing an 'existing' resource
In order to refer to an existing resource, we will either need to expose a `reference` function overload, or to create a new function to take in the type definition and unique identifying properties:

```
[reference(
createObject('type', 'aad:application@1.0', 'provider', 'AAD', 'version', '0.1')
variables('aadProviderConfig'),
createObject('uniqueName', 'myAadApp')
)]
```
> **Q**: Is it practical to overload `reference()`, or do we need a brand new function?

> **NOTE**: At some point, we may want to think about expressing `existing` in the ARM JSON language, as hand-writing reference statements could be very verbose and error-prone.

## Deployment service mechanics
### Call-flows for typical Deployment operations
> **NOTE**: The C# method call / JSONRPC wrapper is assumed in all of the below examples, but dropped for conciseness.

#### Validate (Pre-flight) & What-If
1. For each `$ExtensibilityProvider` resource, ARM invokes the following request on the appropriate provider method:
```json
// ARM -> PROVIDER (resource/previewSave)
{
"type": ...,
"import": ...,
"properties": {
...
}
}
```
> **NOTE**: The contract must provide a means of signalling a value which "is currently unknown" both in the request and response
1. On success:
```json
// PROVIDER -> ARM
{
// The 'preview' of the resource body
"properties": {
...
}
}
```
1. On error:
```json
// PROVIDER -> ARM
{
"errors": [
{ "path": "$.body.someProp", "code": ..., "message": ... }
...
]
}
```
> **Q**: How do we handle preflight/what-if for sets of dependent resources? E.g. evaluating a servicePrincipal for an application that has not yet been created.

#### Deployment (PUT)
1. For each `$ExtensibilityProvider` resource, ARM invokes the following request on the appropriate provider method:
```json
// ARM -> PROVIDER (resource/save)
{
"type": ...,
"import": ...,
"properties": {
...
}
}
```.
1. On success:
```json
// PROVIDER -> ARM
{
// The saved resource body
"properties": {
...
}
}
```
1. On error:
```json
// PROVIDER -> ARM
{
"errors": [
{ "path": "$.body.someProp", "code": ..., "message": ... }
...
]
}
```

#### Deployment (GET)
1. `reference()` requests for 'existing' resources must be translated into a Get on the provider, relying on the fact that the 'identifying' values are present in the get body:
```json
// ARM -> PROVIDER (resource/get)
{
"type": ...,
"import": ...,
"properties": {
"uniqueName": "..."
}
}
```.
1. On success:
```json
// PROVIDER -> ARM
{
// The resource body
"properties": {
...
}
}
```

#### Telemetry/Deployment logs
The Deployment service will log information sufficient to uniquely identify individual resource deployments (types, and identifying property), but no other information from request/response bodies. This information will be used to ensure a deployment graph can be accurately reconstructed from logs.

## Dependency Callouts
1. OBO support from AAD:
* Support for data-plane OBO tokens.
* The ability to refresh OBO tokens on-demand.
1. Packaging/distribution model (for Phase 2).
1. Symbolic name support in Bicep & ARM JSON.
1. Supporting data-plane operations may well lead to a large increase in overall template size. This may necessitate work to increase this limit.

## Other Notes
### Testing & authoring guidelines
As part of extension authoring, we will provide a testing framework along with authoring guidelines, to make it straightforward to verify:
* Idempotency
* Deletion
* Errors

### Deletion
Although the deployment engine does not currently utilize deletion outside of 'complete mode' for tracked resources, it may become a requirement with the work on Stacks.

### Existing feature compatibility
* Export Template: This feature would require the ability for a provider to list 'all' of the resources under a particular scope, which has not been specced out here.
* Resource `scope`, `parent`, nested resources: These are all ARM-specific concepts, and will have no parallel for extensible resources.

Start with the Bicep.Types format linked in the client-side contract, then trace the Bicep CLI/IDE and deployment-engine entry points described in the proposal. Compare the import, type-string, ResourceData, and JSONRPC sections with current behavior; done would require an agreed Phase 1 design and coordinated implementation across authoring, ARM JSON, and deployment mechanics.

Written by the indexing model from the issue text.