RSS feed parsing drops every item when the feed has a single <item> or a single <edgar:xbrlFile>
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 214
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
Noticed while writing tests for #63. Filing separately as suggested there, since it is an independent bug.
### Summary
`xmltodict` collapses a single-element list into a plain dict rather than a one-element list. `parse_rss_feed_data` assumes both `item` and `edgar:xbrlFile` are always lists, so when either has exactly one element the code iterates over dict **keys** (strings) and every item fails to parse.
The per-item `except` in `parse_rss_feed_data` catches the exception and prints a skip message, so the command still exits 0 and writes a header-only CSV. The failure is silent in the sense that matters: **no rows, no error status.**
### Affected lines
- [`rss.py` L151-152](https://github.com/bellingcat/EDGAR/blob/main/src/edgar_tool/rss.py#L151-L152) — `items = xmltodict.parse(...)["rss"]["channel"]["item"]`, then `for i in items`
- [`rss.py` L126-128](https://github.com/bellingcat/EDGAR/blob/main/src/edgar_tool/rss.py#L126-L128) — `files_urls = safe_get(..., "edgar:xbrlFile")`, then `[f.get("@edgar:url") for f in files_urls]`
### Reproduction
Feeding `parse_rss_feed_data` three synthetic feeds that differ only in element counts:
```
A. two items, two xbrlFile each (control)
rows parsed: 2
B. ONE item, two xbrlFile (single )
rows parsed: 0
swallowed: occurred while parsing RSS feed item, skipping it: ("string indices must be integers, not 'str'",)
C. two items, ONE xbrlFile each (single )
rows parsed: 0
swallowed: occurred while parsing RSS feed item, skipping it: ("'str' object has no attribute 'get'",)
```
### Impact
Latent rather than user-visible against `xbrlrss.all.xml`, which normally carries many items each with several files — which is presumably why it has not surfaced. It becomes reachable if the feed is ever quiet enough to return one item, if a filing has exactly one XBRL file, or for anyone reusing `parse_rss_feed_data` against a narrower feed (see #51 re: library use).
Note the existing `unpack_singleton_list` helper deliberately does the *opposite* collapse on the way out, so the one-element case is clearly anticipated elsewhere in the codebase.
### Suggested fix
`xmltodict` has a built-in for exactly this:
```python
xmltodict.parse(response.content, force_list=("item", "edgar:xbrlFile"))
```
Confirmed on the single-element XML above — `item` comes back as a 1-element `list` and `edgar:xbrlFile` as a `list`, so both loops behave. No change needed at the call sites.
Happy to open a PR if that approach looks right.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/edgar_tool/rss.py at parse_rss_feed_data, especially lines 126-152, and reproduce the three synthetic feed cases described in the issue. Verify that feeds with one item or one edgar:xbrlFile produce parsed rows rather than a header-only CSV or skipped-item errors, while the two-item control case remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 86/100