QuantConnect / QuantConnect/Lean
Implement default custom data json reader
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 21.7k
- Forks
- 5.3k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 34
Description
Expected Behavior
Either the entire file contents or a proper json fragment is sent to the Reader method or the documentation needs to be updated here to not show the formatted json fragment like that. Hopefully, that's not the final solution. For it to work with the current design, the json needs to be:
[
{ "Date": "20170704", "Symbols": ["SPY", "QQQ", "FB", "AAPL", "IWM"] },
{ "Date": "20170706", "Symbols": ["QQQ", "AAPL", "IWM", "FB", "GOOGL"] },
...
{ "Date": "20170801", "Symbols": ["QQQ", "FB", "AAPL", "IWM", "GOOGL"] },
{ "Date": "20170802", "Symbols": ["QQQ", "IWM", "FB", "BAC", "GOOGL"] }
]
Actual Behavior
Data is sent one line at a time.
Potential Solution
It seems like CollectionSubscriptionDataSourceReader does try to read data one line a time as indicated here. It may be nice to introduce a new FileFormat (i.e. Json, JsonArray, JsonObject) or add a new field to SubscriptionDataSource to some how indicate the whole file should be read.
Reproducing the Problem
Try using algo and data class from here.
System Information
windows, visual studio
Checklist
- I have completely filled out this template
- I have confirmed that this issue exists on the current
masterbranch - I have confirmed that this is not a duplicate issue by searching issues
- I have provided detailed steps to reproduce the issue
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with CollectionSubscriptionDataSourceReader.cs around line 79 and reproduce the issue using the algorithm and data class from the linked JSON format example. Trace whether the Reader receives one line or the complete file, then compare that behavior with the documentation example. Done means the supported input format is consistent between the reader and the documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, json
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100