catalyst-cooperative / catalyst-cooperative/mozilla-sec-eia
Fix erroneous columns in basic 10k extraction
- Dominant language
- Jupyter Notebook
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
### Overview
During the basic 10K extraction there are some column names that get added to the output like `]fiscal_year_end`, `]irs_number`, `]state_of_incorporation`, instead of `fiscal_year_end`, `irs_number`, and `state_of_incorporation`. While I haven't dug into this yet, this is probably cropping up from a bad field name in the raw basic 10k text itself (an added `[` in front of the field name). I think this can be fixed by just stripping special characters from the field names, or defining the schema/field names ahead of time and enforcing that extracted field names fit into these preset fields.
### Success Criteria
How will we know that we're done?
* [ ] Extracted field names fit into a standard set of columns
```[tasklist]
### Next steps
* [ ] Set standardized field names in basic 10K output table
* [ ] Add cleaning to field names to enforce that standardized field names are met
* [ ] Remove the `_remove_weird_sec_cols` method from the SEC output processing module
```
Contributor guide
Research direction
Start by tracing the basic 10K extraction into the SEC output processing module and inspect `_remove_weird_sec_cols`, along with the raw field names that produce the leading `]`. Define the standard basic 10K output columns, ensure extracted names are cleaned against them, and confirm that the unwanted method can be removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100