Api: order paging surface discards the total count, has no iterator, and is hard to use from research

Summary

Api.ReadBacktestOrders / Api.ReadLiveOrders are the only sanctioned way to get an algorithm's full order stream into a research notebook, and they're difficult to use for exactly that. Working through a "reconcile every fill against my own audit trail" task in a notebook, the paging surface cost roughly an hour of trial and error before producing a single usable row — and every obstacle is fixable in this repo.

public List<ApiOrderResponse> ReadBacktestOrders(int projectId, string backtestId, int start = 0, int end = 100)  // Api/Api.cs:420
public List<ApiOrderResponse> ReadLiveOrders(int projectId, int start = 0, int end = 100)                         // Api/Api.cs:691

Everything below applies identically to both — same wrapper, same element type, same paging contract. Three concrete asks, then a few adjacent things worth cleaning up while someone is in this file.

---

(a) The total order count is fetched, then thrown away

OrdersResponseWrapper carries it, and the doc comment says exactly what it's for:

/// Returns the total order collection length, not only the amount we are sending here
[JsonProperty(PropertyName = "length")]
public int Length { get; set; }

_{Common/Orders/OrdersResponseWrapper.cs:31}

Both methods end in MakeRequestOrThrow<OrdersResponseWrapper>(request, …).Orders, dropping the wrapper on the floor. grep -rn "OrdersResponseWrapper" --include=*.cs over the whole repo returns exactly three hits: the class declaration and those two return statements. Length is deserialized and never read by anything.

The consequence for a caller is that there is no way to answer "how many orders are there?" or "am I done?" other than requesting pages until a short one comes back. You can't size a progress bar, you can't pre-allocate, you can't decide up front whether this is a 3-page job or a 250-page job — which is precisely the decision that determines whether you should be using this API at all.

The insights equivalent already does the right thing and returns InsightResponse with its Length intact (Api/Api.cs:537, 823), as does ReadLiveLogs with LiveLog.Length (Api/Api.cs:759). Orders is the odd one out.

Ask: return the wrapper (or an overload that does), so Length reaches the caller. If the List<ApiOrderResponse> return type needs to stay for compatibility, an overload or an out int total is fine — the point is that the number already crossed the wire and should not be discarded.

(b) There is no paging helper — every caller hand-rolls the loop

To read N orders today you must already know three undocumented-in-Python things: that start/end are indices, that end - start must be ≤ 100, and that you detect the end by a short page (see (a)). So every caller writes the same while loop, and gets it subtly wrong the first few times.

A researcher's first instinct is that a streaming helper exists — the literal first attempt in the session that prompted this issue was:

'Api' object has no attribute 'ReadBacktestOrdersIter'

That guess is the API that should exist. Combined with (a) it's a handful of lines:

public IEnumerable<ApiOrderResponse> ReadBacktestOrdersIterator(int projectId, string backtestId)
{
    var start = 0;
    while (true)
    {
        var response = /* ...request page [start, start + 100)... */;
        foreach (var order in response.Orders) { yield return order; }
        start += response.Orders.Count;
        if (response.Orders.Count == 0 || start >= response.Length) { yield break; }
    }
}

Ask: ship an iterator/generator form for both backtest and live orders that pages internally at the maximum window and terminates on Length. This is the single change that turns "247 pages, too many to pull by hand" into three lines in a notebook.

(c) ApiOrderResponse is hostile from Python

public class ApiOrderResponse : StringRepresentation
{
    public Symbol Symbol { get; set; }
    public Order Order { get; set; }
    public List<SerializedOrderEvent> Events { get; set; }
}

_{Common/Orders/OrdersResponseWrapper.cs:45}

Three surfaces of the same problem, all observed live:

1. The obvious fields aren't where you'd look. id, tag, type, status, quantity are one level down on .Order; the type only exposes Symbol/Order/Events. Every other rendering of an order in the platform puts id and tag at the top level, so the first eight attempts in that session were 'ApiOrderResponse' object has no attribute 'id', followed by …has no attribute 'tag'. Pass-through properties (Id, Tag, or at minimum a documented note) would remove the entire class of error.
2. It can't be pickled, so results can't be cached to disk or carried between notebook cells: cannot pickle 'ApiOrderResponse' object, and cannot pickle 'OrderDirection' object for the nested enum. Any multi-minute fetch has to be re-run from scratch on every kernel restart.
3. There's no DataFrame path. 'ApiOrderResponse' object is not iterable, and there's no to_dataframe() / data_frame property — unlike essentially every other research-facing collection in this repo (History, Indicator, …). Anyone reconciling orders wants a frame, and currently has to write the flattening loop themselves, which requires having already solved (1).

Ask: expose the common order fields directly, make the type picklable, and give the order collection a data_frame accessor consistent with the rest of the research surface.

---

While we're here

A survey of the paged read methods in Api/Api.cs turned up a few more things:

1. Four paged reads, three different meanings for start/end, inconsistent enforcement.

method	window params	unit	cap	validated?	total returned?
ReadBacktestOrders / ReadLiveOrders	start, end	order index	100	no	no
ReadBacktestInsights / ReadLiveInsights	start, end	insight index	100	yes, throws	yes
ReadLiveLogs	startLine, endLine	log line number	250	yes, throws	yes
ReadBacktestChart / ReadLiveChart	start, end	UTC seconds timestamp	—	n/a	n/a

A caller who learns start/end from one of these learns the wrong thing about the next. Charts taking a timestamp under the same parameter names as orders taking an index is a genuine trap — a reasonable guess is that orders wants dates too, and the resulting error (argument 3 ('start') expected int, got datetime.date) doesn't tell you it's an offset. Renaming the order/insight ones to startIndex/endIndex (keeping the old names as overloads) would make each of these self-describing.

2. Orders is the only paged read with no window validation, and it has a latent negative-window bug. Insights guards both the size and the unset-end case:

var diff = end - start;
if (diff > 100) { throw new ArgumentException($"The difference between the start and end index of the insights must be smaller than 100, but it was {diff}."); }
else if (end == 0) { end = start + 100; }

_{Api/Api.cs:540-548}

Orders has neither, while defaulting end = 100. So the natural "just move the cursor" call —

api.read_backtest_orders(project_id, backtest_id, start=500)   # end defaults to 100

— sends start=500, end=100, a negative window, with no client-side complaint. Orders should get the same guard, including the end == 0 → start + 100 convenience, so that passing only start does the obvious thing.

3. The start/end XML docs on orders are misleading.

/// <param name="start">Starting index of the orders to be fetched. Required if end > 100</param>
/// <param name="end">Last index of the orders to be fetched. Note that end - start must be less than 100</param>

_{Api/Api.cs:415-416}

"Required if end > 100" describes a condition that can't be satisfied given the ≤ 100 window rule on the next line, and neither line survives into the Python signature, which is where most readers meet this method.

4. api exists in research and nothing says so. Research/start.py:37 already does:

api = Initializer.GetSystemHandlers().Api

and the notebook templates %run ../start.py, so an authenticated Api instance is in the kernel namespace from the first cell. But BasicQuantBookTemplate.ipynb never mentions it — the template covers QuantBook, history and indicators only. The predictable result is that people construct their own and fail authentication (No method matches given arguments for initialize: ()) while a working one is already bound. A markdown cell plus a commented example in the template is a one-line fix and makes (a)–(c) above actually reachable.

Why this is worth doing

Each item is individually small, and together they're the difference between "pull the order stream into a notebook and compute over it" being a three-line operation and being an hour of guesswork that ends in someone paging by hand. (b) on its own removes most of the pain; (a) is a prerequisite for doing (b) correctly; (c) is what makes the rows usable once you have them.

Happy to open PRs for any subset — (a) + (b) together are the natural first one.

Start in Api/Api.cs with ReadBacktestOrders and ReadLiveOrders, then inspect Common/Orders/OrdersResponseWrapper.cs and compare the existing insights and live-log paging surfaces. Review Research/start.py and BasicQuantBookTemplate.ipynb for the notebook entry point. Done means the requested order count, paging, usability, validation, documentation, and research-template improvements are defined and covered without discarding the response metadata.