Automatic documentation generation for azd extensions

## Summary

azd core has a documentation generation pipeline (`cli/azd/docs/docgen.go`) that produces a unified Markdown CLI reference, published to `MicrosoftDocs/azure-dev-docs-pr` during each release. We need an equivalent story for **azd extensions**, so that each extension can automatically generate and publish a command reference.

## Current state — how core azd docgen works

1. **`cli/azd/docs/docgen.go`** — a standalone `main` package that calls `azd.NewRootCmd(true, nil, nil)` to build the Cobra command tree with static help text (no runtime state). It walks the tree recursively with `genMarkdownFile()`, emitting a single `azd.md` with front-matter, headings per command, flags, examples, code fences, "See also" cross-links, and "Back to top" navigation.

2. **ADO pipeline** (`eng/pipelines/templates/stages/build-and-test.yml`) runs `go run docgen.go` from `cli/azd/docs/`, copies the output into a `docs` pipeline artifact.

3. **Publish step** (`eng/pipelines/templates/steps/publish-cli-docs.yml`) clones `azure-sdk/azure-dev-docs-pr`, copies `azd.md` → `articles/azure-developer-cli/reference.md`, and opens a PR to `MicrosoftDocs/azure-dev-docs-pr`.

### Key coupling to core azd

- `docgen.go` **imports `azd/cmd`** directly to get the root Cobra command — this is a Go compile-time dependency on core azd's entire command tree.
- Extensions don't expose their Cobra tree in the same way; instead they use `azdext.NewExtensionRootCommand()` + `azdext.NewMetadataCommand()` which can output a structured JSON metadata blob.
- The markdown formatting logic (front-matter, code fences, placeholder escaping, link mapping, "See also" sections) is generic and could work with any Cobra tree — it doesn't depend on azd-specific types beyond `*cobra.Command`.

### What extensions already provide

Every extension that declares the `metadata` capability already has:
- A `metadata` subcommand (via `azdext.NewMetadataCommand`) that outputs full JSON with command names, short/long descriptions, usage lines, examples, flags (name, type, default, valid values, hidden, deprecated), and subcommands.
- A Cobra command tree internally (via `azdext.NewExtensionRootCommand`).

---

## Options

### Option A: Extend core `docgen.go` to accept an extension binary path

Add a CLI flag to the existing `docgen.go` so it can be pointed at an extension binary:

```bash
go run docgen.go # existing behavior (core azd)
go run docgen.go -extension ./my-extension # new: generate docs for extension
```

**How it would work:**
- Run ` metadata` to get the JSON command metadata.
- Reconstruct a Cobra tree from the metadata (or parse the JSON directly).
- Feed it through the existing `genMarkdownFile()` with a customizable front-matter template.

**Pros:**
- Single doc-generation tool for the entire ecosystem — one place to fix formatting bugs.
- Leverages the existing, battle-tested markdown generation logic.
- Extensions don't need any new code; they already have `metadata`.

**Cons:**
- Couples extension doc generation to the core azd repo and its Go module. Extension teams must clone/reference the azd repo to generate docs.
- The metadata JSON intentionally omits inherited flags and "See also" relationships — the reconstructed Cobra tree may be lossy compared to the real one.
- Requires the extension binary to be pre-built (or `go run`-able) at doc-generation time.
- Front-matter and formatting may need to differ per extension (different docs repo, different author, etc.).

---

### Option B: `azd x docgen` command in the developer extension

Add a `docgen` subcommand to `azd x` (the extension developer toolkit) that generates docs for the extension in the current directory:

```bash
cd cli/azd/extensions/azure.ai.agents
azd x docgen # generates md/ reference
azd x docgen --output ./docs # custom output dir
```

**How it would work:**
- `azd x docgen` builds the extension, runs ` metadata`, converts the JSON metadata tree to Markdown using shared formatting logic (extracted from `docgen.go` or reimplemented in `azdext`).
- Output format and front-matter are configured via `extension.yaml` fields (e.g., `docs.title`, `docs.author`, `docs.frontMatter`).

**Pros:**
- Natural developer experience — extension authors already use `azd x build`, `azd x test`, etc.
- No dependency on core azd's Go module; the formatting logic lives in `azdext` or a shared package.
- Works for any extension (first-party or third-party) without cloning the azd repo.
- Front-matter and output structure can be configured per extension via `extension.yaml`.

**Cons:**
- Requires implementing (or extracting) the markdown generation logic into the `azdext` package or the developer extension.
- Requires `azd` to be installed to generate docs (may complicate CI for standalone extension repos).
- The `azd x` developer extension is not yet distributed widely; external extension authors may not have it.

---

### Option C: Code-generate a `docgen.go` into the extension

`azd x` generates a self-contained `docs/docgen.go` file inside the extension folder, similar to how the core azd one works — importing the extension's own `cmd` package:

```bash
azd x init-docgen # scaffolds docs/docgen.go + CI pipeline
```

**How it would work:**
- Generate a `docs/docgen.go` that imports the extension's `internal/cmd` (or wherever the root command is) and calls `genMarkdownFile()` from a shared library.
- Optionally scaffold a GitHub Actions workflow (`.github/workflows/publish-docs.yml`) that runs `go run docs/docgen.go` and opens a PR to the extension's docs repo.

**Pros:**
- Each extension is fully self-contained — no dependency on `azd` or `azd x` at doc-generation time.
- Extension teams own their pipeline and can customize freely.
- Works identically to how core azd does it — familiar pattern.

**Cons:**
- Code duplication: every extension gets its own copy of the markdown formatting logic (unless a shared Go library is published and imported).
- Scaffold drift: if the markdown format changes, all extensions need to regenerate or manually update.
- Requires the extension's `cmd` package to be importable (some extensions may have complex init requirements).
- More setup burden per extension (each needs its own docs repo, pipeline config, etc.).

---

### Option D: Shared `azdext/docgen` library + per-extension `docgen.go` (hybrid)

Extract the markdown generation logic from `cli/azd/docs/docgen.go` into a reusable library package (e.g., `pkg/azdext/docgen`). Each extension writes a minimal `docs/docgen.go` that imports this library:

```go
package main

import (
"github.com/azure/azure-dev/cli/azd/pkg/azdext/docgen"
"my-extension/internal/cmd"
)

func main() {
root := cmd.NewRootCommand()
docgen.Generate(root, docgen.Options{
Title: "Azure AI Agents Extension Reference",
Author: "...",
})
}
```

**How it would work:**
- Extract `genMarkdownFile`, `genMarkdownCustom`, `addCodeFencesToSampleCommands`, `escapePlaceholders`, etc. into `pkg/azdext/docgen` as exported functions.
- Publish the azd Go module with semver tags (already done via `cli/azd/vX.Y.Z` tags).
- Each extension adds a thin `docs/docgen.go` + optionally a CI workflow.

**Pros:**
- Single source of truth for formatting logic — updates propagate via Go module versioning.
- Each extension controls its own front-matter, output path, and CI pipeline.
- Minimal boilerplate per extension.
- Works with the existing Go module versioning strategy.

**Cons:**
- Extensions take a Go module dependency on core azd's `pkg/azdext/docgen` — version coupling.
- Still requires per-extension CI setup (though a template could be provided).
- The shared library needs to be generic enough to handle different command tree shapes.

---

## Recommendation

**Option B (`azd x docgen`)** for the best developer experience, with **Option D (shared library)** as a companion for extensions that need standalone CI.

Option B gives the immediate win: extension developers already use `azd x` and the metadata infrastructure is already there. Option D provides a clean path for extensions in separate repos that need to generate docs in their own CI without depending on `azd` being installed.

## CI / Publishing considerations

Each extension may publish docs to a different repo. The publish step should be configurable:
- Target repo (e.g., `MicrosoftDocs/azure-ai-agents-docs-pr`)
- Target path within the repo
- Branch naming convention
- PR title template

This could be driven by `extension.yaml` fields or a separate `docs.yaml` config.

## References

- Core docgen: `cli/azd/docs/docgen.go`
- ADO build step: `eng/pipelines/templates/stages/build-and-test.yml:188-194`
- ADO publish step: `eng/pipelines/templates/steps/publish-cli-docs.yml`
- Extension metadata generator: `cli/azd/pkg/azdext/metadata_generator.go`
- Extension metadata command: `cli/azd/pkg/azdext/extension_commands.go:44-61`
- Extension metadata types: `cli/azd/pkg/extensions/metadata.go`