Remove EFS references from Terraform AWS ECS infrastructure (and the scopes.yml EFS-delivery mechanism)

# Issue Summary: Remove EFS references from the Terraform AWS ECS infrastructure

*Source Issue: [agentic-community/mcp-gateway-registry#452](https://github.com/agentic-community/mcp-gateway-registry/issues/452)* *Fetched: 2026-06-21* *Status: open*

This document captures sub-issue B, derived from splitting GitHub issue #452 ("Remove deprecated functionality: FAISS embeddings, file backend, and EFS references"). Sub-issue B is scoped to the removal of Elastic File System (EFS) references from the Terraform AWS ECS stack, plus the removal of the scopes.yml EFS-delivery mechanism (the build and infrastructure artifacts that ship `scopes.yml` onto the EFS `auth_config` mount). The companion sub-issue (FAISS embeddings plus file backend removal in the Python code) is tracked separately and is independent of this work.

### Relationship to sub-issue A (the scopes.yml code/runtime half)

The scopes.yml work is split across two sub-issues:

- **Sub-issue A (companion, separate)** owns the code and runtime loader half: `registry/common/scopes_loader.py`, `scripts/load-scopes.py`, the scopes manager, the canonical `scopes.yml` files, and Docker Compose scopes mounts. This sub-issue B does NOT modify any of those.
- **Sub-issue B (this document)** owns the EFS-delivery half: the build and infrastructure artifacts whose only job is to put `scopes.yml` onto the EFS `auth_config` mount so a file-backend auth server could read it. On the supported `documentdb` backend nothing reads that file (scopes come from DocumentDB), so the delivery mechanism is dead weight.

Where the two halves touch (for example, `load-scopes.py` is referenced by the EFS-delivery scripts), this document cross-references sub-issue A rather than duplicating its scope.

## Issue Metadata

| Field | Value |
|-------|-------|
| Parent issue | #452 |
| Labels (parent) | technical-debt, architecture |
| Milestone (target for this sub-issue) | 1.24.8 |
| Parent milestone | Parking lot (#9) |
| State | open |

## Original Problem Statement

Issue #452 frames EFS as infrastructure that "was used for the file-based backend which is being deprecated," and proposes removing EFS along with FAISS and the file backend. The premise is that EFS persistence becomes unnecessary once the file backend is gone.

## Requirements Extracted

### From Issue Body

- Remove EFS references from the Terraform AWS ECS infrastructure under `terraform/aws-ecs/`.
- Treat EFS as deprecated alongside the file backend.

### Corrected Premise (from codebase investigation)

The investigation contradicts the issue's premise in two ways, and these corrections define the real scope.

1. **EFS does NOT store FAISS indexes or file-backend registry data.** Those live in container-local `/app/registry/servers/`. The registry ECS service already removed its EFS volumes (comments in `ecs-services.tf` around lines 1586 and 1636 state "EFS volumes removed - registry now uses ephemeral storage and DocumentDB for persistence", with `mountPoints = []` and `volume = {}`). The registry is already EFS-free. So removing the file backend does not by itself remove EFS.

2. **EFS is STILL actively mounted by two other services.** The EFS file system, six access points, mount targets, and NFS security group are all live in `terraform/aws-ecs/modules/mcp-gateway/storage.tf`, and two services still attach EFS volumes:
- **auth-server** mounts EFS twice: access point `logs` to container path `/app/logs`, and access point `auth_config` to container path `/efs/auth_config`. The auth server task sets `SCOPES_CONFIG_PATH = /efs/auth_config/auth_config/scopes.yml`.
- **mcpgw** mounts EFS once: access point `mcpgw_data` to container path `/app/data`.

Therefore the real scope is not a simple delete. It is: re-home or retire each of the three live EFS consumers, then delete the EFS module, all six access points, the mount targets, the NFS security group, the egress rule, the two services' volume and mountPoint blocks, the EFS variables, and the EFS outputs.

### Clarified Through Codebase Analysis

- **scopes.yml is vestigial at runtime on the default deployment.** The auth server loads scopes via `reload_scopes_config()` (`registry/common/scopes_loader.py`), which branches on `STORAGE_BACKEND`. For any MongoDB backend (documentdb, mongodb-ce, mongodb, mongodb-atlas) it calls `load_scopes_from_repository()` and reads from DocumentDB. `SCOPES_CONFIG_PATH` is read ONLY in the file-backend branch (`load_scopes_from_yaml(os.getenv("SCOPES_CONFIG_PATH"))`). The reference deployment sets `storage_backend = "documentdb"` (`terraform/aws-ecs/terraform.tfvars.example` line 264), so the auth server never reads scopes.yml from the EFS mount on the supported AWS deployment. The `SCOPES_CONFIG_PATH` env var and the `auth_config` mount are dead config for ECS.

- **Default scopes are seeded into DocumentDB from JSON, not from the EFS scopes.yml.** All MongoDB backends bootstrap the admin scope from JSON seed files at init time, independent of any EFS scopes.yml:
- terraform/documentdb: `scripts/init-documentdb-indexes.py` `_load_default_scopes()` reads `scripts/registry-admins.json` (called from the script main at line 705, invoked by `run-documentdb-init.sh`).
- Docker Compose mongodb-ce: `scripts/init-mongodb-ce.py` `_load_default_scopes()` reads the same `registry-admins.json`.
- Helm / EKS: the `mongodb-configure` job runs the same seeding logic (`charts/mongodb-configure/templates/configmap.yaml` `_load_default_scopes`).

Because seeding is JSON-based and runs at init time, copying `scopes.yml` to EFS contributes nothing to admin bootstrap on a MongoDB backend. The EFS-delivery mechanism is therefore redundant for seeding as well as for runtime reads.

- **The scopes.yml EFS-delivery mechanism is a set of build and infra artifacts with no runtime consumer on documentdb.** It comprises: a busybox image (`docker/Dockerfile.scopes-init`) that copies `auth_server/scopes.yml` to `/mnt/scopes.yml`; the `mcp-gateway-scopes-init` ECR repository and CodeBuild build entry; a runtime-created `scopes-init` ECS task (`run-scopes-init-task.sh`, invoked by `post-deployment-setup.sh` only in the legacy EFS mode); a CDK `scopes-loader` Lambda (`infra/lambda/scopes-loader/index.py`) that copies the file to the EFS `auth_config` mount; and verification helpers in `service_mgmt.sh` that `cat /app/scopes.yml` inside the container and check a host copy. All of these exist to populate or verify the EFS `auth_config` mount that the documentdb auth server never reads.

- **Updated decision: the CDK scopes-loader Lambda is RETIRED ENTIRELY, not partially.** The Lambda did TWO things: (a) copied `scopes.yml` to the EFS `auth_config` mount (the EFS-delivery part) and (b) upserted UI-Scope group-mapping documents into the DocumentDB `mcp_scopes_` collection (the parity bridge over the JSON seeder, which historically seeded only `registry-admins`). An earlier version of this design preserved (b) and removed only (a). That is now superseded. With the companion sub-issue A adding a JSON seed for `federation-service` and the other defaults (`mcp-registry-admin`, `mcp-servers-unrestricted/*`) already baked into the JSON seed set, all real default scopes are now seeded by `init-documentdb-indexes.py` from JSON. The Lambda's reason to exist (bridging the parity gap) is gone, so the entire Lambda is retired: the EFS copy, the YAML-based DocumentDB upsert, the `SCOPES_YML` synth-time inlining, the `scopes-loader.ts` staging, the `AwsCustomResource` invocation, and the CDK EFS `auth_config` wiring all go. The line-of-business and public example scopes (`registry-users-lob1`, `registry-users-lob2`, `public-mcp-users`) are treated as examples and are not seeded. See `lld.md` for the exact resolution and the precondition below.
- **In-scope deliverable: wire the JSON seeder into the CDK deploy, then full retire.** Retiring the Lambda entirely is only safe if the CDK deploy path actually runs the JSON seeder (`init-documentdb-indexes.py`, or an equivalent JSON-based seeder) with the FULL set of JSON seed files, including the new `federation-service.json` from sub-issue A. Investigation shows this is NOT currently the case: the CDK path (`infra/scripts/deploy.sh` -> `infra/scripts/post-deploy.sh`) seeds scopes via `load-scopes.py --scopes-file /app/config/scopes.yml` (a YAML loader, `post-deploy.sh` `_load_scopes()` at line ~291) plus the ScopesLoader Lambda; it does NOT invoke `init-documentdb-indexes.py` at all. `init-documentdb-indexes.py` is invoked only by the terraform path (`run-documentdb-init.sh`, `init-documentdb.sh`); in `infra/` it appears only inside a code comment in `registry-service-stack.ts`. The chosen approach (user decision) is "wire the JSON seeder into CDK, then full retire": sub-issue B wires `init-documentdb-indexes.py` (with the full JSON seed set including `federation-service.json`) into the CDK deploy as a replacement for both the Lambda's DocumentDB upsert and the YAML-based `load-scopes.py` seeding in `post-deploy.sh` `_load_scopes()` (~line 291), then retires the Lambda entirely. This is an in-scope work item for sub-issue B, not an external precondition or open question. See `lld.md` and `review.md` for the implementation steps.
- **The auth-server `/app/logs` EFS mount is also vestigial.** `APP_LOG_DIR` defaults to empty (`var.app_log_dir` default `""`), which the application resolves to `/var/log/containers/ai-registry`, not `/app/logs`. Container stdout is already shipped to CloudWatch via `enable_cloudwatch_logging = true` on the task (log group `/ecs/-auth-server`, 30-day retention). Nothing depends on the EFS `/app/logs` mount.
- **mcpgw writes nothing durable to `/app/data`.** `servers/mcpgw/server.py` runs the FastMCP app with `stateless_http=True`, and `servers/mcpgw/logging_setup.py` logs to `/var/log/containers/ai-registry`. There is no read or write to `/app/data` in the mcpgw source. The mount is vestigial.

## Existing Design Elements

The issue body does not contain a technical removal plan. The plan is established in the accompanying `lld.md`.

## Acceptance Criteria

### From Issue

- [ ] EFS references removed from `terraform/aws-ecs/`.

### Additional (Established)

- [ ] `terraform validate` passes after removal.
- [ ] `terraform plan` shows only destruction of EFS-related resources (file system, six access points, mount targets, NFS security group, egress rule) and the two task-definition revisions that drop EFS volumes/mountPoints. No unintended changes to unrelated resources.
- [ ] `terraform state list | grep -i efs` returns nothing after apply.
- [ ] The auth-server task definition no longer references EFS volumes, mountPoints, the `auth_config` access point, or `SCOPES_CONFIG_PATH`.
- [ ] The mcpgw task definition no longer references the EFS volume or the `/app/data` mountPoint.
- [ ] `efs_throughput_mode` and `efs_provisioned_throughput` variables are removed from the module.
- [ ] `efs_id`, `efs_arn`, `efs_access_points` outputs are removed from the module and the root.
- [ ] After deploy: auth-server reaches RUNNING, login and authorization still work (scopes sourced from DocumentDB), and the MCP proxy canary (`ai-registry-tools` / mcpgw) still resolves through the full nginx -> auth_request -> mcp-proxy chain.

### Additional (scopes.yml EFS-delivery removal)

- [ ] `docker/Dockerfile.scopes-init` is deleted.
- [ ] The `mcp-gateway-scopes-init` ECR repository and its CodeBuild build/pull entries are removed from `terraform/aws-ecs/codebuild.tf` (lines ~33, ~219, ~270).
- [ ] `terraform/aws-ecs/scripts/run-scopes-init-task.sh` is deleted, and its caller in `post-deployment-setup.sh` (the EFS-mode branch, lines ~549-568) is removed so post-deploy uses only the documentdb init path.
- [ ] The commented-out `load-scopes.py --scopes-file /app/config/scopes.yml` reference and its explanatory comment are removed from `terraform/aws-ecs/scripts/run-documentdb-init.sh` (lines ~215-227), leaving the active JSON-seeding path intact.
- [ ] The CDK `scopes-loader` Lambda is retired in full: `infra/lambda/scopes-loader/index.py` (the whole directory), the `AwsCustomResource` invocation and `ScopesLoader` instantiation in `infra/lib/registry/registry-service-stack.ts` (lines ~460-478), the `scopes-loader.ts` construct (`infra/lib/registry/constructs/scopes-loader.ts`), the `SCOPES_YML` synth-time inlining, and the EFS `auth_config` wiring passed to it are all deleted. `infra/docs/CDK-INFRASTRUCTURE.md` is updated to drop the Lambda description entirely.
- [ ] Committed scope for the above (the "wire JSON seeder into CDK, then full retire" approach): sub-issue B wires the JSON seeder (`init-documentdb-indexes.py` or equivalent) with the full set of JSON seed files (including `federation-service.json` from sub-issue A) into the CDK deploy, replacing both the Lambda's DocumentDB upsert and the YAML-based `load-scopes.py` seeding in `post-deploy.sh` `_load_scopes()` (~line 291). The Lambda retirement follows the wiring within the same sub-issue. This is an in-scope deliverable, not a gate on external work.
- [ ] `federation-service.json` (the new JSON seed file added in sub-issue A) is present and included in the JSON seed set consumed by `init-documentdb-indexes.py`.
- [ ] `service_mgmt.sh` scopes.yml verification (`verify_scopes_yml()` ~line 139 and its callers ~627, ~719, ~1147-1205, plus the create-group/delete-group help text ~952-953) is removed or replaced with a DocumentDB scope-repository check (decision documented in `lld.md`).
- [ ] `terraform/aws-ecs/terraform.tfvars.example` line ~400 comment "Groups must exist in scopes.yml group_mappings" is updated to reference DocumentDB scope group mappings.
- [ ] A fresh terraform/documentdb deploy seeds the `registry-admins` admin scope and admin login works WITHOUT the `scopes-init` task ever running.
- [ ] After the Lambda is retired, a fresh CDK documentdb deploy still has ALL default scopes (including `federation-service`, `mcp-registry-admin`, `mcp-servers-unrestricted/*`) present in DocumentDB, sourced purely from JSON seeding via `init-documentdb-indexes.py`.

## Scope

In scope (Terraform AWS ECS only):

1. Re-home or confirm-unused: auth-server logs (EFS `logs`), auth-server scopes.yml (EFS `auth_config` plus `SCOPES_CONFIG_PATH`), mcpgw data (EFS `mcpgw_data`).
2. Delete the EFS volume and mountPoint blocks from the auth-server and mcpgw task definitions in `ecs-services.tf`.
3. Delete the EFS module, all six access points, mount targets, NFS security group, and the egress rule in `storage.tf`.
4. Delete the `efs_throughput_mode` and `efs_provisioned_throughput` module variables.
5. Delete the `efs_id`, `efs_arn`, `efs_access_points` outputs at module and root level.
6. Remove the scopes.yml EFS-delivery artifacts: `docker/Dockerfile.scopes-init`; the `mcp-gateway-scopes-init` ECR repo and CodeBuild build/pull entries; `run-scopes-init-task.sh` and its `post-deployment-setup.sh` caller; the dead `load-scopes.py` reference in `run-documentdb-init.sh`; the CDK `scopes-loader` Lambda in full (the `index.py` directory, the `AwsCustomResource` / `ScopesLoader` instantiation, the `scopes-loader.ts` construct, the `SCOPES_YML` synth-time inlining, and the EFS `auth_config` wiring); the `service_mgmt.sh` scopes.yml verification; and the `terraform.tfvars.example` scopes.yml comment. The Lambda retirement follows the JSON-seeder wiring (item 7) within this sub-issue.
7. Wire the JSON seeder into the CDK deploy (in-scope deliverable, the "wire then retire" approach): the CDK path runs `init-documentdb-indexes.py` (or an equivalent JSON seeder) with the full JSON seed set, including `federation-service.json` from sub-issue A, replacing both the Lambda's DocumentDB upsert and the YAML-based `load-scopes.py` seeding in `post-deploy.sh` `_load_scopes()` (~line 291). The Lambda is then retired entirely. This is committed work in sub-issue B, not an external precondition.

## Constraints

- **Docker Compose and Helm are unaffected by the EFS and scopes.yml-delivery removal.** Neither surface uses the AWS EFS module, and neither delivers scopes.yml via EFS. Docker Compose mongodb-ce seeds default scopes from `registry-admins.json` via `init-mongodb-ce.py`, and Helm seeds them via the `mongodb-configure` job. Removing the EFS scopes-delivery artifacts touches only the Terraform AWS ECS and CDK paths. This is stated explicitly so reviewers do not look for Compose or Helm regressions.
- **Backwards compatibility with operator-run init.** Operators may seed scopes through their own init job or the registry API. Removing the `scopes-init` delivery task does not assume our JSON seeding is the only path. The supported terraform/documentdb path seeds the admin scope via `init-documentdb-indexes.py`, and any additional scopes can be created through the API or an operator-supplied job. The removal narrows the maintained surface; it does not block alternative seeding.

## Out of Scope

- FAISS embeddings removal and file backend removal in Python code (companion sub-issue).
- Any change to the registry ECS service (already EFS-free).
- Docker Compose and Helm deployments (they do not use the AWS EFS module or EFS scopes delivery; verify quickly but do not modify unless a leftover EFS reference is found).
- Changing the application's scopes-loading code. This work removes the dead `SCOPES_CONFIG_PATH` ECS wiring and the EFS-delivery artifacts only; the file-backend code path remains for non-AWS / local file deployments. The scopes loader, `load-scopes.py`, the scopes manager, and the canonical `scopes.yml` files are owned by sub-issue A.
- Changing the JSON seed files themselves (for example creating `federation-service.json`). That is owned by sub-issue A. This sub-issue depends on those JSON seeds existing but does not author them.

## Key Decisions from Discussion

| Decision | Context | Decided By |
|----------|---------|------------|
| Treat all three EFS uses as vestigial on the documentdb deployment and remove rather than re-platform to S3/Secrets Manager | Investigation shows none are read/written at runtime on the supported backend | This design (pending maintainer confirmation) |
| Keep the file-backend Python branch intact | Local/file deployments still rely on `SCOPES_CONFIG_PATH`; only the ECS wiring is dead | This design |
| Add a JSON seed for `federation-service` and bake all real default scopes into init code | With `federation-service.json` (sub-issue A) plus the already-JSON-seeded `mcp-registry-admin` and `mcp-servers-unrestricted/*`, `init-documentdb-indexes.py` seeds all real defaults from JSON. The line-of-business and public example scopes are not seeded. | User decision |
| Retire the CDK scopes-loader Lambda ENTIRELY (both the EFS copy and the YAML-based DocumentDB upsert) | Once all real default scopes are JSON-seeded, the Lambda's parity-bridge reason to exist is gone. This supersedes the earlier "preserve the DocumentDB upsert" decision. | User decision |
| Wire the JSON seeder into the CDK deploy, then full retire the Lambda (both steps in sub-issue B) | The CDK path does not currently run `init-documentdb-indexes.py` (it relied on the Lambda plus YAML `load-scopes.py`). The chosen approach makes the JSON-seeder wiring an in-scope deliverable that lands before the Lambda retirement, so non-admin defaults are never dropped. | User decision |
| Remove (do not migrate) the `scopes-init` ECS task and its build artifacts | The task exists only to write scopes.yml to EFS in the legacy EFS mode; documentdb deploys use `run-documentdb-init.sh` and never invoke it | This design |

## Open Questions Resolved

| Question | Resolution |
|----------|------------|
| Does the auth server read scopes.yml from disk at runtime? | Only when `STORAGE_BACKEND` is `file`. The ECS reference deployment uses `documentdb`, so it reads from DocumentDB and never touches the EFS scopes.yml. |
| How does scopes.yml get onto the EFS access point? | No Terraform-managed `.tf` resource writes scopes.yml to EFS. Delivery happens out of band: in the legacy EFS mode, `post-deployment-setup.sh` invokes `run-scopes-init-task.sh`, which registers and runs a `scopes-init` ECS task from the busybox image to copy the file. In the CDK path, the `scopes-loader` Lambda copies it. On the documentdb deployment neither delivery is needed because nothing reads the file, so both are being removed (the Lambda's DocumentDB seeding is kept). |
| Does the `scopes-init` task or the CDK Lambda also seed DocumentDB, or only copy to EFS? | The busybox `scopes-init` task (terraform path) ONLY copies to EFS; it does not touch DocumentDB. The CDK `scopes-loader` Lambda did BOTH: copied to EFS AND upserted UI-Scope group docs into DocumentDB. Under the updated decision the entire Lambda is retired (both jobs), contingent on the JSON-seeder precondition below. |
| Will the terraform/documentdb path still seed scopes after removal? | Yes. `run-documentdb-init.sh` runs `init-documentdb-indexes.py`, whose `_load_default_scopes()` seeds the default scopes from the JSON seed set (`registry-admins.json` plus the new `federation-service.json` and the other JSON-seeded defaults). This is independent of the EFS scopes.yml. |
| Will the CDK documentdb path still seed scopes after the Lambda is retired? | Yes, once the in-scope wiring lands. The CDK path does NOT currently run `init-documentdb-indexes.py`; it seeds via `load-scopes.py --scopes-file /app/config/scopes.yml` (YAML) plus the Lambda. Sub-issue B wires `init-documentdb-indexes.py` (with all JSON seeds, including `federation-service.json`) into the CDK deploy, replacing both the YAML seeding and the Lambda's DocumentDB upsert, then retires the Lambda. This is committed work in sub-issue B (the "wire then retire" approach), not an open question. |
| What does mcpgw write to `/app/data`? | Nothing. The server is `stateless_http=True` and logs to `/var/log/containers/ai-registry`. |
| Where do auth logs go without EFS? | CloudWatch (already configured via `enable_cloudwatch_logging`) for stdout, plus the in-container default log dir `/var/log/containers/ai-registry`. |

## Dependencies

- None blocking. This sub-issue is independent of the FAISS/file-backend Python removal. It can ship before, after, or alongside it.

## Notes

EFS deletion is destructive and irreversible for any data on the file system. Although the investigation shows the three live mounts are vestigial, the destroy ordering still matters: AWS requires all mount targets to be deleted before the file system, and Terraform must sequence this correctly. The Rollout Plan in `lld.md` calls this out and recommends a snapshot/inventory step before apply.