GSA / GSA/data.gov

Harvest notification email says job succeeded even when it errored (ExtractExternalException)

Open
#6,338 0 comments 0 reactions 1 assignee Claimed by @Aguiardavidm View on GitHub
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
4h 56m
Merged PRs (30d)
5

Description

## Summary
Ran a harvest job for `healthdata-gov` in development against `https://hhs.github.io/OCDO-Public-Data-Inventory/prod/data.json` (blocked by the dev egress proxy - see #6336 - not the point of this ticket). The job errored during extraction, but the notification email says the opposite.

## Repro
- Job log: `ExtractExternalException` - "healthdata-gov https://hhs.github.io/OCDO-Public-Data-Inventory/prod/data.json failed to extract harvest source. exiting :: ProxyError(...403 Forbidden...)"
- Job page: https://harvest-dev.data.gov/harvest_job/6231ef49-7fd5-4cb2-8a31-50d5bb1d830c
- Email received (subject "Harvest Job Completed"):
> A harvest job has been successfully completed.
> - Organization: U.S. Department of Health & Human Services
> - Harvest source: healthdata-gov
> ...
> Records Added: 0 / Updated: 0 / Deleted: 0 / Unchanged: 0 / Errored: 0 / Warned: 0 / Validated: 0

All-zero counts with "successfully completed" language gives no indication anything went wrong.

## Root cause
`datagov-harvester/harvester/harvest.py`:
- `run_full_harvest()` catches `ExtractExternalException`/`ExtractInternalException`, calls `finish_job_with_status("error")`, and returns early (harvest.py:806-813).
- `harvest_job_starter()` then unconditionally calls `harvest_source.report()` regardless of that error (harvest.py:1691).
- `report()` correctly avoids overwriting the job status back to "complete" when it's already "error" (harvest.py:836-841), but nothing else in `report()`/`send_notification_emails()` accounts for that status - the email subject/body are hardcoded to success language (harvest.py:869, 892) regardless of job status, and the job never got far enough to populate `job_results`, so every count is 0.

## Suggested fix
In `send_notification_emails` (or `report`), branch the subject/body on the job's actual status - e.g. "Harvest Job Failed" with the exception message, instead of the hardcoded success text, when status is "error".

## Impact
Source POCs (and the shared `SMTP_CONFIG['recipient']`) get a false "all good, nothing happened" email on hard extraction failures, so a broken/misconfigured harvest source can go unnoticed indefinitely.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.