QuantConnect / QuantConnect/Lean

Expand `IDataQueueHandler` for `DataSource` consumers

Open
#9,272 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
21.7k
Forks
5.3k
Avg merge
2d 22h
Merged PRs (30d)
34

Description

Problem

IDataQueueHandler is currently too rigid for long-running DataSource consumers. When a breaking change is needed, we risk interface churn and downstream breakage.

Additionally, unhandled exceptions inside LongRunning tasks are swallowed by the task infrastructure and do not propagate to the main thread, which leaves the DataSource running in an undefined or stalled state.

At the moment, there is no reliable mechanism for a DataSource to notify Lean that a fatal error has occurred and that it should be terminated.

Expected Behavior
  • A DataSource should be able to:
    • Surface fatal exceptions to Lean
    • Terminate cleanly when an unrecoverable error occurs
  • Lean should receive a clear error signal (e.g. via BrokerageMessageEvent) when a DataSource fails
  • Future extensions to IDataQueueHandler should not require breaking interface changes
Actual Behavior
  • Exceptions thrown inside LongRunning tasks are swallowed
  • The DataSource continues running (or appears to) without notifying Lean
  • No XXXMessageEvent is emitted to indicate failure
  • Consumers cannot safely extend behavior without modifying the core interface
// Exception thrown here is swallowed
_dataReceiverTask = new Task(
    () => MonitorDataReceiverConnection(token),
    token,
    TaskCreationOptions.LongRunning);

_dataReceiverTask.Start();
Potential Solution
  1. Introduce an extensible abstraction
    1. Add a base class (e.g. DataQueueHandlerBase) that implements IDataQueueHandler
    2. Allow new virtual hooks without breaking consumers
  2. Create an extended interface
    1. e.g. IExtendedDataQueueHandler : IDataQueueHandler
    2. Optional adoption by advanced DataSource implementations
  3. Add a fatal error signaling mechanism
    1. Allow IDataQueueHandler / base class to raise a Lean event on failure:
OnMessage(new BrokerageMessageEvent(
    BrokerageMessageType.Error,
    "DataSource",
    "Fatal error occurred. DataSource is terminating."));
  1. Lean should react by stopping the DataSource deterministically
  2. Centralize exception handling
    1. Catch exceptions inside LongRunning tasks
    2. Route them through the error event instead of letting them disappear
Benefits
  • Prevents silent failures in live data sources
  • Improves observability and debuggability
  • Enables safer evolution of IDataQueueHandler
  • Reduces breaking changes for downstream consumers
Checklist
  • I have completely filled out this template
  • I have confirmed that this issue exists on the current master branch
  • I have confirmed that this is not a duplicate issue by searching issues

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating IDataQueueHandler, its DataSource consumers, the LongRunning task path, and BrokerageMessageEvent handling. Trace how exceptions and shutdown currently flow, then define how extensibility, fatal-error signaling, and deterministic DataSource termination should be verified; the issue provides no files or tests to target.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.