dataset with bad json gets harvested
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 200
- Avg merge
- 20h 10m
- Merged PRs (30d)
- 6
Description
This dataset always generates an error during the OpenSearch indexing.
https://catalog-beta.data.gov/dataset/4aa5d8cd-f4ac-4c3b-88d3-78c3cf495aa9
## Index error log:
```
Warning: 1 dataset(s) failed to index in this batch.
{'dataset_id': '4aa5d8cd-f4ac-4c3b-88d3-78c3cf495aa9', 'status_code': 400, 'error_type': 'mapper_parsing_exception', 'error_reason': 'failed to parse', 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'field name cannot be an empty string'}}
{'index': {'_index': 'datasets', '_id': '4aa5d8cd-f4ac-4c3b-88d3-78c3cf495aa9', 'status': 400, 'error': {'type': 'mapper_parsing_exception', 'reason': 'failed to parse', 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'field name cannot be an empty string'}}}}
```
## Dataset detail page:
## Source JSON:
## We need to:
- manually delete the record from DB
- bad json should not be harvested
Contributor guide
Research direction
No repository files, tests, or entry points are named. Start with the linked dataset, source JSON, and OpenSearch indexing error, then trace the harvesting path that accepts this record. Done means the malformed dataset is no longer harvested and the existing bad record has been manually removed from the database.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100