QuantConnect / QuantConnect/Documentation
Morningstar US Fundamentals: state that SecurityReference status/delisting fields (and classification) are current values, not point-in-time
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 261
- Forks
- 191
- Avg merge
- 16h 28m
- Merged PRs (30d)
- 56
Description
Summary
The US Fundamental Data (Morningstar) pages do not state that the SecurityReference status fields are current reference values stamped onto every historical daily snapshot, not point-in-time values. Users qualifying point-in-time universe selection read share_class_status on a historical date and treat D (Deactive) as "delisted as of that date", which silently excludes every later-delisted company at formation time.
What we observed (LEAN 2.5.0.0.18057, cloud backtest, history[Fundamental](symbol, 2005-01-01, 2024-01-01))
| ticker | rows | security_reference.share_class_status |
security_reference.delisting_date |
|---|---|---|---|
| CELG (delisted 2019-11-21) | 3748 (2005-01-04 .. 2019-11-21) | D on every row |
2019-11-26 on every row, including 2005 |
| TWTR (delisted 2022-10-28) | 2260 (2013-11-08 .. 2022-10-29) | D on every row |
2022-10-28 on every row |
| AAPL | 4781 | A on every row |
none |
| XOM | 4781 | A on every row |
none |
The explicit FundamentalService.Get<string>(date, sid, FundamentalProperty.SecurityReference_ShareClassStatus) route returns the same: CELG is D on 2010-06-30, 2015-06-30, 2019-06-28 and 2019-11-20, and null from 2019-11-21 (no snapshot after the last trading day). delisting_reason, company_status, is_primary_share (TWTR: False on every row) and exchange_id behave the same way. This is consistent with the earlier finding that the sector / industry codes are today's assignment backfilled across history (see QuantConnect/Lean#9777's discussion of provenance).
Field definition in Lean: Common/Data/Fundamental/Generated/SecurityReference.cs (ShareClassStatus, Morningstar DataId 1014: "(A) Active, (D) Deactive, (I) Inactive, or (O) Obsolete").
Proposed documentation change
On 03 Writing Algorithms/14 Datasets/04 Morningstar/01 US Fundamental Data (Introduction and/or Data Point Attributes), add a short note along the lines of:
The financial statement and earnings-report values are as-originally-reported per snapshot date. The reference and classification fields (
security_reference.share_class_status,delisting_date,delisting_reason,company_status,is_primary_share,exchange_id, and theasset_classificationsector/industry codes) are Morningstar's current values and are the same on every historical snapshot. Do not use them to decide historical membership; a company delisted in 2019 readsDwith its 2019 delisting date on its 2005 rows. For point-in-time membership use the daily universe itself (history[Fundamentals]/universe_history, which include later-delisted companies) and theDelistingevents for cessation dates.
Optionally cross-link this from the Fundamental Universes page, which is where users build formation-date selections.
Open question for the data team
Whether Morningstar supplies effective-dated status / delisting history that could be written into the daily files instead. If so, this becomes a data change rather than a docs note.
Intercom conversation: 215475854522100.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with 03 Writing Algorithms/14 Datasets/04 Morningstar/01 US Fundamental Data and review the Fundamental Universes page for the best cross-link location. Check field definitions in Common/Data/Fundamental/Generated/SecurityReference.cs and the examples in this issue. Done means the documentation clearly distinguishes current reference/classification values from historical financial values and explains how to select point-in-time membership; confirm whether the open data-team question changes this from a docs-only task.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 76/100