GSA / GSA/data.gov

Deploy validator traffic to a separate Cloud Foundry application

Open
#6,293 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

The public DCAT-US schema validator currently runs in the same Cloud Foundry application as the Harvester administrative interface.

Validator-related paths include:

```
- /validate
- /validate/
- /api/validate
- /api/v1/validate
```

## Problem

Keeping the validator and admin application together has several drawbacks:

- Validator requests consume the same CPU, memory, and Gunicorn workers as administrative operations.
- Large or expensive validation requests could reduce the availability of Harvester administration.
- The server-side URL-fetching feature increases the security exposure of the admin application.
- The validator cannot be scaled, restarted, or monitored independently.

## Proposed approach

Deploy the existing codebase as an additional Cloud Foundry application named `datagov-harvest-validator`.

The existing datagov-harvest application will remain functionally unchanged and continue to contain the validator routes. Nginx will direct public validator traffic to the new application.

```
harvest.data.gov
|
datagov-harvest-proxy
|
+-- validator routes
| -> datagov-harvest-validator
|
+-- all other routes
-> datagov-harvest
```

This provides runtime and capacity isolation without requiring an immediate restructuring of the Flask application.

## Things to consider

- Give the validator application its own internal route, instances, memory, and scaling configuration.
- Use a startup command that does not run database migrations.
- Route `/validate, /validate/, /api/validate, and /api/v1/validate` through nginx to the new application.
- Add the required proxy-to-validator network policy and deployment workflow steps.
- Keep the validator routes in the main application to provide a simple rollback path.
- Because the new application initially loads the full Flask application, it may still require existing service bindings and secrets. A validator-only application factory can be considered later.
- Consider additional protections for public validator traffic, including request timeouts, redirect validation, egress hostname restrictions, and rate limiting.

Contributor guide

Open the contributing guide

Research direction

Start by locating the nginx routing configuration and deployment workflow, then trace the existing validator paths and Cloud Foundry application settings. Done means validator routes reach datagov-harvest-validator with independent resources, the proxy-to-validator network policy and deployment steps work, startup avoids migrations, and non-validator routes remain on datagov-harvest.

Written by the indexing model from the issue text.

Assessment

Tech stack
nginx, python
Domain
cloud, devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.