Issues with types.json and Bicep extensibility

# Issues with types.json and Bicep extensibility

This is a catalog of design issues/concerns found using the `types.json` format and tooling to [teach Bicep about Kubernetes](https://github.com/rynowak/bicep/tree/radius-compiler-kubernetes).

Individual items mentioned here represent design issues or challenges - basically all of the ways that the Bicep compiler e2e is coupled to ARM's types. Not all of these have to be solved, I'm erring on the side of including design issues even if there is a solution. In general the coupling between Bicep and ARM is spread across many layers currently - from the `types.json` code to the compiler's semantic diagnostics and code generation.

## Structure and generation of types.json

`types.json` is the FFI format for ingesting types into the Bicep compiler. It's designed as a non-human-readable/writable representation of the Bicep type system. In constrast the APIs Bicep code wants to interface with are generally described using JSON-Schema or the the Swagger/OpenAPI variant of JSON-Schema.

There's therefore an impedence mismatch - JSON-schema can express a wide variety of complex schemas that the Bicep compiler cannot understand. The process of generating a `types.json` for an API means resolving this impedence mismatch and mapping the set of JSON-schema types for an API into the set of Bicep types. This process will likely be different for every API we want to map into Bicep.

The converter/generator of `types.json` in this repo is built for ARM's REST API conventions, and has deep understanding of the patterns that appear in those OpenAPI documents. When I tried to use the autorest-based converter, I found that I had to re-implement almost all of it just to get to hello world. Dialects of OpenAPI used by different systems are drastically different in practice.

Such a conversion tool is responsible many tasks:

- Parsing OpenAPI/Swagger into an object model (generic)
- Determining which types should be exposed as resources (domain-specific)
- Mapping endpoints and types to Bicep resource type semantics: resource type name, api-version, resource scope, parent types (domain-specific)
- Structural conversion from JSON-Schema to Bicep types (domain specific)
- Generation of the `types.json` document and index structure from Bicep types (generic)

In practice for a non-ARM system all of the steps named as *domain specific* will need to be written from scratch or highly customized. Systems extend OpenAPI in proprietary ways, or use features sets that don't overlap with ARM.

Some examples:

- Determining which types should be exposed as resources: *look for `x-kubernetes-kind`*
- Mapping endpoints and types to Bicep resource type semantics: *look for other `x-kubernetes-...` annotations, scrape parameter lists to determine scope*
- Structural conversion from JSON-Schema to Bicep types: *Kubernetes objects tend to follow different conventions from ARM, will be described in detail later*

I'd strongly suggest turning `types.json` into a fully-documented and human-readable/writable format, and let the extensibility author write their own generatior/mapper. Another clickstop would be to allow Bicep to read JSON-schema with annotations that are specific to Bicep's type system. Then the scope of a conversion tool is limited to trimming down the original document into just the relevant types and adding annotations to describe the Bicep behavior.

## Coupling to ARM's conventions

`types.json` does not fully describe the behavior of a type once it gets inside the compiler. There are important semantic details and behaviors that are hardcoded in the compiler rather than data-driven by the format.

For reference there's a set of *well-known* properties defined:

- here: https://github.com/Azure/bicep/blob/main/src/Bicep.Core/LanguageConstants.cs#L82
- here: https://github.com/Azure/bicep/blob/main/src/Bicep.Core/LanguageConstants.cs#L97
- and here: https://github.com/Azure/bicep/blob/main/src/Bicep.Core/LanguageConstants.cs#L113

These property are given special status by the Bicep compiler, and compiler features rely on these properties appearing at the top level of an object for correctness. For a motivating example the `name` property is required to be loop-variant, and required to be unique for a type in the same template. Many places in code will discard a resource that does not define name. However in Kubernetes the *actual* property with those semantics is `.metadata.name`.

For each of the special behaviors that can be granted to a property it should be possible to indicate in `types.json` with property flags to which properties they apply.

A short list of things that bake in ARM's conventions.

- The *name* equivalent.
- The *scope* equivalent.
- The *parent* equivalent.
- The equivalent of *type* and *apiVersion*. (`kind` in Kubernetes)
- Parent/Child relationships are based on the format of an ARM fully-qualified type rather than metadata.
- Ability to define a deploy-time-constant property

To detail this further - compare the [property flags supported by `types.json`](https://github.com/Azure/bicep-types-az/blob/main/src/autorest.bicep/src/types.ts#L57) and the [property flags supported by Bicep](https://github.com/Azure/bicep/blob/main/src/Bicep.Core/TypeSystem/TypePropertyFlags.cs#L8)

## Features that are not extensible

- Functions & Decorators have no mechanism for extensibility via `types.json`. A function can only be implemented in side the C# part of a type provider.
- Fallback type (the type that is used when the type cannot be resolved) is not extensible via `types.json`. A fallback type can only be implemented in side the C# part of a type provider.
- Scopes are not extensible by any mechanism, and the compiler has deep coupling to ARM's scope concepts. Kubernetes has both `namespace` and `cluster` as meaningful scopes.

## Missing features in object structure

Kubernetes objects largely don't use discriminated unions in the same way ARM resources do - they use a different pattern (`oneOf`) to express a *choice*. You can read more about their conventions [here](https://kubernetes.io/blog/2019/06/20/crd-structural-schema/).

Providing a good tooling experience for other systems means supporting some consensus set of features they rely-on for validation/structure.

An example of this pattern using different sources to set environment variables. Only one of either `value` or `valueFrom` can be used.

```yaml
apiVersion: v1
kind: Pod
metadata:
name: envar-demo
spec:
containers:
- name: envar-demo-container
image: gcr.io/google-samples/node-hello:1.0
env:
- name: DEMO_GREETING
value: "Hello from the environment" # using .value to set an environment variable
- name: DEMO_FAREWELL
valueFrom: # using .valueFrom to set an environment variable
secretKeyRef:
name: mysecret
key: goodbye
```

In ARM this would be expressed similar to:

```
resource foo '...' = {
...
properties: {
....
env: [
{
name: DEMO_GREETING
value: {
kind: 'static'
value: 'Hello from the environment'
}
}
{
name: DEMO_FAREWELL
value: {
kind: 'secret'
name: mysecret
key: goodbye
}
}
]
}
}
```

Start by reading LanguageConstants.cs, TypePropertyFlags.cs, and the property flags in types.ts to map the compiler's ARM-specific assumptions. Then trace the type-provider and types.json entry points mentioned in the issue. Done would require an agreed, implementable extensibility design covering the listed conventions, scopes, functions, fallback types, and schema patterns.

Written by the indexing model from the issue text.