microsoft / microsoft/simplechat

Add versioned, hash-verified SSRF destination registry for Azure, third-party, and custom environments

Open
#1,344 0 comments 0 reactions 0 assignees View on GitHub
enhancement security_improvement
Dominant language
Python
Stars
152
Forks
116
Avg merge
7h 7m
Merged PRs (30d)
122

Description

## Summary

Introduce a versioned, integrity-verified **SSRF destination registry** that ships with SimpleChat as data rather than as scattered hard-coded strings. The registry enumerates the fully-qualified domain names (FQDNs) and URL suffixes that SimpleChat legitimately talks to, tagged by cloud and by service, so that outbound-request policy can be evaluated against a curated, reviewable, auditable source of truth.

Three registry files are in scope:

| File | Contents | Shipped populated | Integrity-checked |
|---|---|---|---|
| Azure registry | Azure Public (Commercial) and Azure US Government service FQDN suffixes, each tagged with its cloud | Yes | Yes — version + hash |
| Third-party registry | Non-Azure providers SimpleChat can legitimately reach: AWS, GCP, Anthropic, OpenAI, and `genai.mil` | Yes | Yes — version + hash |
| Custom-environment example | The same structure, empty, as a template for sovereign/air-gapped/custom deployments | No — example only | No |

The registry is then surfaced in the Admin Settings **Security** group as a **read-only** card, and a missing/empty custom registry in a custom environment raises both an in-card notice and an admin notification.

## User Value

- **Operators** get a single, explicit answer to "what is SimpleChat allowed to call out to?" instead of that answer being distributed across `config.py` branches, `functions_source_review.py` constants, and `functions_mcp_destinations.py` pattern lists.
- **Government, sovereign, and air-gapped customers** get a supported way to declare their own service endpoints without patching Python. Today a custom cloud deployment has no declarative place to express its egress surface.
- **Security reviewers and approvers** get a small, diffable, hash-pinned artifact to review, rather than needing to re-audit application code to understand egress posture.
- **Admins** get visibility: the Security tab shows what the effective allowed destination set actually is for their environment, including the case where it is dangerously empty.

## Background — what exists today

SSRF defenses already exist, but they are fragmented and none of them are data-driven or versioned:

- `application/single_app/functions_source_review.py` — `normalize_review_url()`, `validate_source_review_url()`, `evaluate_source_review_url_policy()`, plus hard-coded `BLOCKED_HOSTNAMES`, `BLOCKED_HOSTNAME_SUFFIXES`, and `INTERNAL_HOSTNAME_SUFFIXES` constants.
- `application/single_app/functions_mcp_destinations.py` — MCP destination governance with `MCP_UNSAFE_METADATA_IPS`, allow/deny pattern lists, and `evaluate_mcp_destination_policy()` / `assert_mcp_destination_allowed()`.
- `application/single_app/functions_url_access_policy_test.py` — `run_url_access_policy_test()` for the admin-facing policy test helper.
- Admin-entered allow/deny lists: `url_access_allowed_domains`, `url_access_blocked_domains`, `source_review_allowed_domains`, `source_review_blocked_domains`.
- `application/single_app/config.py` — `AZURE_ENVIRONMENT` (`public` | `usgovernment` | `custom`) already branches per-cloud endpoints (authority, Resource Manager, Cognitive Services, Video Indexer, Key Vault domain), and `custom` is expressed only through `CUSTOM_*_URL_VALUE` environment variables.

This issue does not replace those mechanisms. It gives them a shared, versioned data layer to consult.

## Proposed Behavior

### 1. Registry files and format

Add registry data files shipped with the application (location to be settled in design; the existing precedent for shipped JSON data is `application/single_app/mcp_preconfigurations/definitions/` and `application/single_app/static/json/`).

Each entry should carry at minimum:

- The FQDN or DNS suffix (for example `.openai.azure.com`, `.search.windows.net`, `.vault.usgovcloudapi.net`, `api.videoindexer.ai.azure.us`).
- A **cloud tag** (`azure_public` / `azure_usgovernment`, or the third-party provider name) so a single file can carry multiple clouds and consumers can filter.
- The **service** it belongs to, mapped where possible to the admin settings key that configures it, so the card can explain *why* a destination is allowed. Relevant keys include:
`azure_openai_gpt_endpoint`, `azure_openai_embedding_endpoint`, `azure_openai_image_gen_endpoint`, the matching `azure_apim_*_endpoint` variants, `content_safety_endpoint`, `azure_ai_search_endpoint`, `azure_document_intelligence_endpoint`, `azure_content_understanding_endpoint`, `speech_service_endpoint`, `video_indexer_endpoint`, `redis_url`, `front_door_url`, `office_docs_storage_account_url`, `office_docs_storage_account_blob_endpoint`, `video_files_storage_account_url`, `audio_files_storage_account_url`.
- Optional notes/description for reviewer context.

Important: a US Government deployment is **not** restricted to only `azure_usgovernment` entries. Admins in one cloud may legitimately consume services tagged for another. The cloud tag is metadata for filtering and display, not a hard partition.

### 2. Versioning and integrity

The Azure and third-party registries each carry:

- An explicit **registry version** that is independent of the application `VERSION` in `application/single_app/config.py`.
- A recorded **content hash**, so an unreviewed change to the allowed egress surface is detectable.

The custom-environment file is an **example** and is deliberately excluded from hash enforcement, since deployments are expected to edit it.

### 3. CI enforcement

Add a GitHub Actions workflow that fails loudly and explicitly when a registry's content hash changes without the corresponding version and expected-hash being updated in the same change.

Rationale: a Development → Staging PR can carry tens of thousands of lines, and a one-line change to an SSRF allowlist would be trivially lost in that diff. The signal has to be unmissable.

Conventions to follow from existing workflows:

- `.github/workflows/swagger-route-check.yml` and `.github/workflows/xss-sink-check.yml` are the closest models: path-filtered `pull_request` triggers, SHA-pinned actions, and a `scripts/check_*.py` validator invoked from the workflow.
- `.github/workflows/enforce-branch-flow.yml` establishes the `Development` → `Staging` → `main` promotion flow, so the check must at minimum gate PRs into `Staging`.

### 4. Admin Settings — Security tab card

Add a new **read-only** card under the Security group. Tab structure is defined in `application/single_app/admin_settings_nav.py` (the `security` group currently contains `access-roles`, `secrets`, `content-safety`, `session`, and `network`), rendered by `application/single_app/templates/admin_settings.html`.

Behavior:

- This is **not** an enable/disable feature. There is no toggle. The card reports state.
- Show the registry version and hash status.
- Show the effective allowed destination set, filtered and grouped by cloud tag, with the service each entry belongs to.
- When `AZURE_ENVIRONMENT` is `custom`, additionally show the custom registry contents.
- When `AZURE_ENVIRONMENT` is `custom` **and** the custom registry is absent or empty, show an inline warning notice in the card explaining that the deployment has declared no custom destinations.

Any settings passed to this view must go through `sanitize_settings_for_user()` where the route is not admin-scoped, per `.github/instructions/santize_settings_for_frontend_routes.instructions.md`.

### 5. Admin notification for unpopulated custom environments

When running in a custom environment with an empty custom registry, raise a persistent admin notification so it surfaces in the admin notification list rather than only on a page an admin may never open.

Use the existing system in `application/single_app/functions_notifications.py`: `create_notification()` supports role-targeted delivery via `assignment={'roles': ['Admin', ...]}`, and a new entry will be needed in `NOTIFICATION_TYPES`. `KEY_VAULT_SECRET_REMINDER_NOTIFICATION_TYPE` is a good precedent for a config-health notification type.

## Open Questions

These are explicitly unresolved and should be settled during design, not assumed:

1. **Unlisted custom model endpoints.** #1222 adds a Custom model endpoint provider, which means model endpoints will legitimately live outside Azure. If an admin configures a Custom endpoint whose FQDN is in no registry file, what happens? Options include hard deny, deny with an actionable admin remediation path, or warn-and-allow plus an admin notification. Hard deny is the safest default but is operationally harsh given that the shipped registries can only change on a release. The custom registry file is the intended escape hatch — the question is whether an unpopulated escape hatch should be fail-closed or fail-open.
2. **Enforcement vs. advisory in v1.** Should the registry initially be advisory (surfaced and reported, consulted by existing validators as an additional allow source) with enforcement gated behind a follow-up, or enforcing from the start?
3. **Relationship to the existing admin-entered allow/deny lists.** Does the registry union with `url_access_allowed_domains` / `source_review_allowed_domains`, or take precedence, or sit at a different layer entirely?
4. **File location and load path.** Shipped alongside `mcp_preconfigurations/`, under `static/json/`, or a new dedicated directory. Also whether the custom file is read from the image or from a mounted/config-supplied path, given that editing a file inside a container image is not a good operator experience.
5. **Registry scope creep.** Does the registry also cover MCP destinations, OpenAPI action base URLs, and source-review targets, or is it initially scoped to admin-settings service endpoints and model endpoints only?

## Acceptance Criteria

- [ ] Azure registry file exists, enumerating Azure Public (Commercial) and Azure US Government service FQDN suffixes, each tagged with its cloud and its owning service/settings key.
- [ ] Third-party registry file exists covering AWS, GCP, Anthropic, OpenAI, and `genai.mil`, each tagged with its provider.
- [ ] Custom-environment example file exists with the full structure and no populated destinations.
- [ ] Both shipped registries carry an independent version and a recorded content hash; the custom example is excluded from hash enforcement.
- [ ] A GitHub Actions workflow fails a PR — at minimum into `Staging` — when a shipped registry's content hash changes without the version and expected hash being updated, and the failure message is explicit enough to be unmissable in a very large diff.
- [ ] A read-only card appears in the Admin Settings Security group showing registry version, hash status, and the effective destination set grouped by cloud tag and service.
- [ ] The card is not gated behind an enable/disable toggle.
- [ ] A deployment tagged `azure_usgovernment` can still see and use destinations tagged `azure_public`; the cloud tag filters and labels rather than partitions.
- [ ] In a `custom` environment, the card additionally shows the custom registry, and shows an inline warning when it is empty or missing.
- [ ] In a `custom` environment with an empty custom registry, an admin-targeted notification is created through `functions_notifications.py` with a registered notification type.
- [ ] Behavior for an unlisted Custom model endpoint (#1222) is decided, documented, and implemented consistently with that decision.
- [ ] Admin-facing documentation is updated — `docs/admin/security.md` plus feature documentation under `docs/explanation/features/`, and the docs inventory is regenerated per `.github/instructions/docs_coverage.instructions.md`.
- [ ] Operator-facing README/setup guidance explains that custom-environment deployments must populate the custom registry for SSRF policy to be meaningful.
- [ ] Functional tests under `functional_tests/` cover registry parsing/schema validation, hash and version verification, cloud-tag filtering, the empty-custom-registry warning and notification path, and the admin card rendering. Tests use `assert_app_version_at_least()` from `functional_tests/test_support/versioning.py` rather than exact version equality.
- [ ] `VERSION` in `application/single_app/config.py` is incremented and release notes are updated in `docs/explanation/release_notes.md`.

## Notes

- Related: #1222 (Custom model endpoint provider) — the primary driver for Open Question 1, since it moves model traffic outside Azure.
- Related: #1014 (harden outbound token-based MCP actions) — overlapping outbound-destination governance surface; see Open Question 5.
- Current application version at time of filing: `0.260.028`.
- No secrets or credentials belong in any registry file; these are destination names only.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing SSRF and notification entry points in application/single_app/functions_source_review.py, functions_mcp_destinations.py, functions_notifications.py, and config.py, then review the Admin Settings structure in admin_settings_nav.py and admin_settings.html. Compare the path-filtered workflow patterns in .github/workflows/swagger-route-check.yml and xss-sink-check.yml, and inspect the relevant functional_tests and documentation instructions. Done means the registries, verification workflow, Security card, custom-environment notification, tests, and specified documentation are implemented after the open design questions are resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
backend, ci-cd, documentation, security, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.