QuantConnect / QuantConnect/Lean
Expand `IDataQueueHandler` for `DataSource` consumers
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 21.7k
- Forks
- 5.3k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 34
Description
Problem
IDataQueueHandler is currently too rigid for long-running DataSource consumers. When a breaking change is needed, we risk interface churn and downstream breakage.
Additionally, unhandled exceptions inside LongRunning tasks are swallowed by the task infrastructure and do not propagate to the main thread, which leaves the DataSource running in an undefined or stalled state.
At the moment, there is no reliable mechanism for a DataSource to notify Lean that a fatal error has occurred and that it should be terminated.
Expected Behavior
- A
DataSourceshould be able to:- Surface fatal exceptions to Lean
- Terminate cleanly when an unrecoverable error occurs
- Lean should receive a clear error signal (e.g. via
BrokerageMessageEvent) when aDataSourcefails - Future extensions to IDataQueueHandler should not require breaking interface changes
Actual Behavior
- Exceptions thrown inside
LongRunningtasks are swallowed - The
DataSourcecontinues running (or appears to) without notifying Lean - No
XXXMessageEventis emitted to indicate failure - Consumers cannot safely extend behavior without modifying the core interface
// Exception thrown here is swallowed
_dataReceiverTask = new Task(
() => MonitorDataReceiverConnection(token),
token,
TaskCreationOptions.LongRunning);
_dataReceiverTask.Start();
Potential Solution
- Introduce an extensible abstraction
- Add a base class (e.g.
DataQueueHandlerBase) that implementsIDataQueueHandler - Allow new virtual hooks without breaking consumers
- Add a base class (e.g.
- Create an extended interface
- e.g.
IExtendedDataQueueHandler:IDataQueueHandler - Optional adoption by advanced
DataSourceimplementations
- e.g.
- Add a fatal error signaling mechanism
- Allow
IDataQueueHandler/ base class to raise a Lean event on failure:
- Allow
OnMessage(new BrokerageMessageEvent(
BrokerageMessageType.Error,
"DataSource",
"Fatal error occurred. DataSource is terminating."));
- Lean should react by stopping the
DataSourcedeterministically - Centralize exception handling
- Catch exceptions inside
LongRunningtasks - Route them through the error event instead of letting them disappear
- Catch exceptions inside
Benefits
- Prevents silent failures in live data sources
- Improves observability and debuggability
- Enables safer evolution of
IDataQueueHandler - Reduces breaking changes for downstream consumers
Checklist
- I have completely filled out this template
- I have confirmed that this issue exists on the current
masterbranch - I have confirmed that this is not a duplicate issue by searching issues
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating IDataQueueHandler, its DataSource consumers, the LongRunning task path, and BrokerageMessageEvent handling. Trace how exceptions and shutdown currently flow, then define how extensibility, fatal-error signaling, and deterministic DataSource termination should be verified; the issue provides no files or tests to target.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100