bellingcat / bellingcat/EDGAR

RSS feed parsing drops every item when the feed has a single <item> or a single <edgar:xbrlFile>

Open Beginner friendly
#64 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
214
Forks
26
PR merge metrics
No merged PRs in 30d

Description

Noticed while writing tests for #63. Filing separately as suggested there, since it is an independent bug.

### Summary

`xmltodict` collapses a single-element list into a plain dict rather than a one-element list. `parse_rss_feed_data` assumes both `item` and `edgar:xbrlFile` are always lists, so when either has exactly one element the code iterates over dict **keys** (strings) and every item fails to parse.

The per-item `except` in `parse_rss_feed_data` catches the exception and prints a skip message, so the command still exits 0 and writes a header-only CSV. The failure is silent in the sense that matters: **no rows, no error status.**

### Affected lines

- [`rss.py` L151-152](https://github.com/bellingcat/EDGAR/blob/main/src/edgar_tool/rss.py#L151-L152) — `items = xmltodict.parse(...)["rss"]["channel"]["item"]`, then `for i in items`
- [`rss.py` L126-128](https://github.com/bellingcat/EDGAR/blob/main/src/edgar_tool/rss.py#L126-L128) — `files_urls = safe_get(..., "edgar:xbrlFile")`, then `[f.get("@edgar:url") for f in files_urls]`

### Reproduction

Feeding `parse_rss_feed_data` three synthetic feeds that differ only in element counts:

```
A. two items, two xbrlFile each (control)
rows parsed: 2

B. ONE item, two xbrlFile (single )
rows parsed: 0
swallowed: occurred while parsing RSS feed item, skipping it: ("string indices must be integers, not 'str'",)

C. two items, ONE xbrlFile each (single )
rows parsed: 0
swallowed: occurred while parsing RSS feed item, skipping it: ("'str' object has no attribute 'get'",)
```

### Impact

Latent rather than user-visible against `xbrlrss.all.xml`, which normally carries many items each with several files — which is presumably why it has not surfaced. It becomes reachable if the feed is ever quiet enough to return one item, if a filing has exactly one XBRL file, or for anyone reusing `parse_rss_feed_data` against a narrower feed (see #51 re: library use).

Note the existing `unpack_singleton_list` helper deliberately does the *opposite* collapse on the way out, so the one-element case is clearly anticipated elsewhere in the codebase.

### Suggested fix

`xmltodict` has a built-in for exactly this:

```python
xmltodict.parse(response.content, force_list=("item", "edgar:xbrlFile"))
```

Confirmed on the single-element XML above — `item` comes back as a 1-element `list` and `edgar:xbrlFile` as a `list`, so both loops behave. No change needed at the call sites.

Happy to open a PR if that approach looks right.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/edgar_tool/rss.py at parse_rss_feed_data, especially lines 126-152, and reproduce the three synthetic feed cases described in the issue. Verify that feeds with one item or one edgar:xbrlFile produce parsed rows rather than a header-only CSV or skipped-item errors, while the two-item control case remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.