microsoft / microsoft/simplechat
Scale per-row tabular analysis and generated CSV exports to 30,000+ rows
@paullizer is already working on this.
Since Jul 14, 2026.
- Dominant language
- Python
- Stars
- 152
- Forks
- 116
- Avg merge
- 7h 7m
- Merged PRs (30d)
- 122
Description
## Issue
Per-row tabular analysis works for small files but fails to produce the requested exhaustive CSV once the source rows are split across multiple tabular tool pages. The customer scenario asks nine independent questions against every CSV row, requires one output row per source row, then expects an overall summary and any remaining prompt actions.
The attached reproduction succeeds with 10 rows and fails with 300 rows. The design must support 300, 3,000, and 30,000 rows with bounded memory and durable progress; larger files may take longer, but row count must not change correctness.
## Steps to Reproduce
1. Upload `priv-simplechat-2/tabular_issue/simplechat_row_orchestration_dataset_300.csv`.
2. Select the document in chat.
3. Ask SimpleChat to answer nine questions for each row, create a CSV with one output row per source row, then provide an overall analysis.
4. Compare the generated artifact and backend diagnostics with the 10-row fixture `simplechat_row_orchestration_dataset.csv`.
## Expected Behavior
- Every source row is processed exactly once and remains in source order.
- The generated CSV contains exactly one output row per source row with a stable schema and source identity.
- Work proceeds in bounded, checkpointed batches and can resume after worker or model-service interruption.
- The final CSV is committed and attached only after all batches validate.
- The assistant then returns a compact overall analysis and completes remaining requested actions.
- Progress, cancellation, retry, and failure details remain visible for long-running jobs.
## Actual Behavior
- The 300-row `query_tabular_data` result is safely paged into `94 + 95 + 94 + 17` rows.
- Generated-output source selection evaluates each invocation independently instead of treating compatible pages as one exhaustive result.
- The largest page reports `95` returned rows out of `300` total, so export creation logs `Selected source candidate is incomplete; skipping export`.
- The final assistant response may contain a small summary table; the generic assistant-table exporter then saves that summary as a misleading nine-row CSV rather than the requested 300-row deliverable.
## Root Cause
1. `route_backend_chats.py::_build_tabular_generated_output_source_candidate(...)` ranks individual tool invocations and requires one invocation to contain the full result.
2. Tool pagination from `TabularProcessingPlugin._build_tabular_row_page_payload(...)` is correct, but generated-output selection does not coalesce or replay those pages.
3. Durable background export queueing begins only after all source rows are already present and staged as one input-batches blob.
4. Background finalization loads all checkpoint outputs into one Python list before serializing and uploading the artifact.
5. The assistant-table fallback is not the primary fault, but becomes misleading when the exhaustive export was skipped.
## Impact
**P0.** This blocks a customer-required row orchestration workflow and makes output appear complete when it is not. The failure threshold depends on serialized row width rather than a predictable row limit, so even moderately sized files can fail. Current architecture also cannot make a defensible 30,000-row scalability guarantee.
## Required Remediation
- Define a stable row transformation contract and output schema once per run.
- Preserve source identity and row order through every batch.
- Fix the immediate multi-page candidate gap without making the model retrieve all rows into one context.
- Queue large exhaustive transforms from an authorized source/query snapshot rather than from a fully materialized model-tool result.
- Read bounded source windows, checkpoint idempotent model outputs, and resume from the next incomplete window.
- Assemble the final CSV/JSON from ordered checkpoints without consolidating all output rows in memory; use a staged/atomic final artifact commit.
- Revalidate conversation ownership and current workspace authorization when background workers start or resume.
- Keep existing transient retry, scheduler recovery, progress UI, cancellation, and generated-artifact presentation behavior.
- Suppress the assistant summary-table CSV fallback whenever an exhaustive export is queued, running, completed, or failed with an explicit status.
## Acceptance Criteria
- [ ] The 10-row fixture produces exactly 10 ordered output rows.
- [ ] The 300-row fixture produces exactly 300 unique ordered rows from `SC-2001` through `SC-2300`.
- [ ] Synthetic 3,000- and 30,000-row fixtures produce exactly N rows without omissions or duplicates using bounded source and output windows.
- [ ] A worker restart or transient model failure resumes from durable checkpoints without regenerating completed rows.
- [ ] Gaps, overlaps, schema drift, count mismatches, malformed model output, or changed source versions fail explicitly before final artifact commit.
- [ ] Current personal, group, public-workspace, and conversation authorization is revalidated at background execution boundaries.
- [ ] Final CSV and JSON artifacts are valid, correctly quoted/encoded, size-limited, and attached only after complete validation.
- [ ] The generic assistant-table fallback cannot create a partial summary CSV for an active exhaustive export.
- [ ] The assistant provides a compact post-run summary derived from batch summaries without putting all output rows back into model context.
- [ ] Functional tests cover direct, paginated, large, resume, retry, cancellation, authorization, atomic-finalization, and fallback-suppression paths.
- [ ] Relevant feature/fix documentation, release notes, and `application/single_app/config.py` version are updated when implementation lands.
## Implementation Notes
- Reuse the existing durable run, scheduler, checkpoint blobs, progress API/UI, retry classification, and generated chat artifact model in `functions_tabular_generated_exports.py`.
- Do not introduce a second background export subsystem.
- Treat input query/source descriptors as authorization-sensitive. Resolve them in request context and revalidate durable scope at execution time.
- For final storage, prefer ordered Azure Block Blob staging/commit or another atomic equivalent over appending to a user-visible CSV.
- Preserve the existing system prompt rule that one input row must yield exactly one output object.
## Planning Document
`docs/explanation/fixes/TABULAR_ROW_ORCHESTRATION_REMEDIATION_PLAN.md`
## Related Work
- Refs #802: move tabular orchestration out of the chat route.
- Refs #966: audit remaining Tabular Phase 2 improvements.
- Refs #1021: turn-level orchestration and central finalization contracts.
- Closed #893 delivered tabular tool pagination, auto-trim, and projection; this issue addresses the missing exhaustive export orchestration above that layer.
- Existing resilience docs: `LARGE_TABULAR_GENERATED_EXPORT_TIMEOUT_FIX.md`, `TABULAR_BACKGROUND_EXPORT_TRANSIENT_RESUME_FIX.md`, `TABULAR_BACKGROUND_EXPORT_QUEUED_RETRY_RESUME_FIX.md`, and `TABULAR_BACKGROUND_EXPORT_SCHEDULER_RECYCLE_FIX.md`.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.