influxdata / influxdata/ui

Improve Data Explorer Experience when querying Measurements with large number of rows

Open
#6,683 0 comments 0 reactions 0 assignees View on GitHub
kind/bug team/ui
Dominant language
TypeScript
Stars
117
Forks
51
Avg merge
2d 15h
Merged PRs (30d)
4

Description

## About the bug
In MTC IOx, I have a rather large table (> 1,000,000 entries) which generates the following error when attempting to converting the PyArrow Table (query results) to Pandas. The Table in the Data Explorer displays Max. display limit exceeded. Results truncated to 100.21 MB.

In Python it returns:
in pyarrow._flight.check_flight_status() FlightInternalError: Flight returned internal error, with message: Received RST_STREAM with error code 2. gRPC client debug context: UNKNOWN:Error received from peer ipv4:34.196.233.7:443 {grpc_message:"Received RST_STREAM with error code 2", grpc_status:13, created_time:"2023-04-29T10:43:12.588955-05:00"}. Client context: OK

**Steps to reproduce:**
1. Populate a Measurement with more than 1,000,000 (7,712,800 rows) rows within the retention policy. I am using Telegraf with Windows Process.
2. Use Data Explorer to view the Measurement - "select *" to return all fields as if exporting to CSV, etc.
3. Set time range to 30 days or entire retention period
4. Run Query
5. View the Message regarding display limit. "Max. display limit exceeded. Results truncated to 102 MB"

**Expected behavior:**
Would like to have an option to query large table and return all rows. Additional information could be added to the message to communicate with the user.

**Actual behavior:**
Max display limit exceeded. Results truncated to 102 MB, yet the

**Visual Proof:**
Screenshot attached. The CSV export seems to export all 7,712,792 rows.

## About your environment

**Environment info:**

IOx MTC with Data Explorer
Telegraf on Windows 10 writing Process data to IOx namespace.

![MTC-IOx-Proc-Query-Screenshot 2023-04-29 112842 (1)](https://user-images.githubusercontent.com/128092913/235509007-6e5afbf5-7f9c-4e41-9850-908aaf8ff83d.png)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in the Data Explorer code that handles query results and the “Max. display limit exceeded” message, then compare it with the CSV export path, which reportedly returns all rows. Define the user-visible behavior for querying large measurements, including how the display limit is communicated and whether an option can return all rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data, frontend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.