Basekick-Labs / Basekick-Labs/arc

Clients and docs do not surface the truncated / Arc-Stream-Truncated signals

Open
#726 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
677
Forks
53
Avg merge
9h 14m
Merged PRs (30d)
164

Description

Follow-up to #721 and #723. Both added server-side signals that a streamed result was cut short. Nothing on the client side reads them yet, so a user still sees partial data presented as complete.

What now exists on the server:

- `Arc-Stream-Truncated` trailer on Arrow IPC responses, carrying the reason (#722)
- an unsatisfiable Arrow message so an Arrow IPC decode fails loudly rather than returning a short result (#722)
- `"truncated": true` plus `truncation_reason` in the JSON envelope (#723)

What does not read them:

- **Grafana datasource**: `queryJSON` decodes into `map[string]interface{}` and passes it to `JSONToDataFrame`, which reads only `columns` and `data`. A truncated result renders as a normal panel with fewer points, which is exactly the silent-data-loss case these fixes were meant to end. It should surface a query error, or at minimum a panel warning.
- **Python SDK**: the documented `QueryResult` exposes `columns`, `data`, `row_count` only.
- **arcli**: worth checking whether it prints the field.
- **Docs**: the JSON response field table in the API reference and the Python SDK `QueryResult` table are the published schema and do not mention `truncated`. Until they do, a third-party client author has no reason to check it.

The Arrow IPC side is in better shape: a decode failure is unmissable for any client. It is the JSON path, the default and the one Grafana uses, where the signal is purely advisory.

Suggested order:

1. Docs: add `truncated` and `truncation_reason` to the response field table, and the trailer to the Arrow IPC section, with a sentence saying what a client should do.
2. Grafana datasource: treat `truncated` as a query error, or attach a frame notice.
3. Python SDK and arcli: expose the field.

Splitting per repo is fine; this issue is the umbrella.

Contributor guide

Open the contributing guide

Research direction

Start with the API reference JSON response field table and Arrow IPC section, then inspect the Grafana datasource's queryJSON and JSONToDataFrame path, the Python SDK QueryResult, and arcli output handling. Decide how each client should surface truncated and truncation_reason. Done means the documented fields and trailer explain client action, and the affected clients no longer present truncated JSON results as complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, json, python
Domain
api, documentation, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.