GSA / GSA/data.gov

dataset with bad json gets harvested

Open
#5,728 1 comment 0 reactions 0 assignees View on GitHub
bug Harvester OpenSearch
Dominant language
Python
Stars
1.1k
Forks
200
Avg merge
20h 10m
Merged PRs (30d)
6

Description

This dataset always generates an error during the OpenSearch indexing.
https://catalog-beta.data.gov/dataset/4aa5d8cd-f4ac-4c3b-88d3-78c3cf495aa9

## Index error log:
```
Warning: 1 dataset(s) failed to index in this batch.
{'dataset_id': '4aa5d8cd-f4ac-4c3b-88d3-78c3cf495aa9', 'status_code': 400, 'error_type': 'mapper_parsing_exception', 'error_reason': 'failed to parse', 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'field name cannot be an empty string'}}
{'index': {'_index': 'datasets', '_id': '4aa5d8cd-f4ac-4c3b-88d3-78c3cf495aa9', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': 'failed to parse', 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'field name cannot be an empty string'}}}}
```

## Dataset detail page:
Image

## Source JSON:
Image

## We need to:
- manually delete the record from DB
- bad json should not be harvested

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or entry points are named. Start with the linked dataset, source JSON, and OpenSearch indexing error, then trace the harvesting path that accepts this record. Done means the malformed dataset is no longer harvested and the existing bad record has been manually removed from the database.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.