Deploy validator traffic to a separate Cloud Foundry application
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 200
- Avg merge
- 4h 56m
- Merged PRs (30d)
- 5
Description
The public DCAT-US schema validator currently runs in the same Cloud Foundry application as the Harvester administrative interface.
Validator-related paths include:
```
- /validate
- /validate/
- /api/validate
- /api/v1/validate
```
## Problem
Keeping the validator and admin application together has several drawbacks:
- Validator requests consume the same CPU, memory, and Gunicorn workers as administrative operations.
- Large or expensive validation requests could reduce the availability of Harvester administration.
- The server-side URL-fetching feature increases the security exposure of the admin application.
- The validator cannot be scaled, restarted, or monitored independently.
## Proposed approach
Deploy the existing codebase as an additional Cloud Foundry application named `datagov-harvest-validator`.
The existing datagov-harvest application will remain functionally unchanged and continue to contain the validator routes. Nginx will direct public validator traffic to the new application.
```
harvest.data.gov
|
datagov-harvest-proxy
|
+-- validator routes
| -> datagov-harvest-validator
|
+-- all other routes
-> datagov-harvest
```
This provides runtime and capacity isolation without requiring an immediate restructuring of the Flask application.
## Things to consider
- Give the validator application its own internal route, instances, memory, and scaling configuration.
- Use a startup command that does not run database migrations.
- Route `/validate, /validate/, /api/validate, and /api/v1/validate` through nginx to the new application.
- Add the required proxy-to-validator network policy and deployment workflow steps.
- Keep the validator routes in the main application to provide a simple rollback path.
- Because the new application initially loads the full Flask application, it may still require existing service bindings and secrets. A validator-only application factory can be considered later.
- Consider additional protections for public validator traffic, including request timeouts, redirect validation, egress hostname restrictions, and rate limiting.
Contributor guide
Research direction
Start by locating the nginx routing configuration and deployment workflow, then trace the existing validator paths and Cloud Foundry application settings. Done means validator routes reach datagov-harvest-validator with independent resources, the proxy-to-validator network policy and deployment steps work, startup avoids migrations, and non-validator routes remain on datagov-harvest.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nginx, python
- Domain
- cloud, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100