QuantConnect / QuantConnect/Lean

Implement default custom data json reader

Open
#8,874 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

depth feature
Dominant language
C#
Stars
21.7k
Forks
5.3k
Avg merge
2d 22h
Merged PRs (30d)
34

Description

Expected Behavior

Either the entire file contents or a proper json fragment is sent to the Reader method or the documentation needs to be updated here to not show the formatted json fragment like that. Hopefully, that's not the final solution. For it to work with the current design, the json needs to be:

[
    {  "Date": "20170704",   "Symbols": ["SPY", "QQQ", "FB", "AAPL", "IWM"]    },
    { "Date": "20170706",   "Symbols": ["QQQ", "AAPL", "IWM", "FB", "GOOGL"]   },
    ...
    {  "Date": "20170801",   "Symbols": ["QQQ", "FB", "AAPL", "IWM", "GOOGL"]   },
    {  "Date": "20170802",   "Symbols": ["QQQ", "IWM", "FB", "BAC", "GOOGL"]     }
]
Actual Behavior

Data is sent one line at a time.

Potential Solution

It seems like CollectionSubscriptionDataSourceReader does try to read data one line a time as indicated here. It may be nice to introduce a new FileFormat (i.e. Json, JsonArray, JsonObject) or add a new field to SubscriptionDataSource to some how indicate the whole file should be read.

Reproducing the Problem

Try using algo and data class from here.

System Information

windows, visual studio

Checklist
  • I have completely filled out this template
  • I have confirmed that this issue exists on the current master branch
  • I have confirmed that this is not a duplicate issue by searching issues
  • I have provided detailed steps to reproduce the issue

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CollectionSubscriptionDataSourceReader.cs around line 79 and reproduce the issue using the algorithm and data class from the linked JSON format example. Trace whether the Reader receives one line or the complete file, then compare that behavior with the documentation example. Done means the supported input format is consistent between the reader and the documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, json
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.