apache / apache/superset

NULL values are visualized inconsistently across different plugins

Open
#43,547 1 comment 0 reactions 0 assignees View on GitHub
#bug viz:charts:boxplot viz:charts:bubble viz:charts:chord viz:charts:histogram viz:charts:radar viz:charts:treemap
Dominant language
Python
Stars
74.8k
Forks
18.3k
Avg merge
2d 5h
Merged PRs (30d)
685

Description

### Bug description

**NULL values are visualized inconsistently across different plugins, which can be misleading for users.**

There are three main inconsistencies, listed in order of severity, in my opinion:
1. NULL value dimensions are not shown in certain charts
2. NULL_STRING is not shown next to NULL value dimensions in certain charts
3. NULL_STRING is not shown in tooltips for NULL value dimensions in certain charts

Perhaps these are three separate bugs with three separate fixes, but I wanted to report it in one sweep as general NULL visualization issues as I think it needs a general refactor/consolidation.

**1. NULL value dimensions are not shown in certain charts**
In certain charts, NULL value dimensions are not shown entirely. I've spotted this when comparing a box plot and bar chart using the same dataset and configuration, notice 7 boxes vs 8 bars:

Image

Image

Histograms seem to have the same issue.

I think it's because `aggregate.py `calls `df.groupby(by=groupby)` without `dropna=False`. Pandas defaults to dropping NaN keys, so the whole group vanishes before it reaches the frontend.

**2. NULL_STRING is not shown next to NULL value dimensions in certain charts**
NULL_STRING is not shown next to NULL value dimensions in certain charts, I've spotted it in Chord diagrams, graph charts, and partition charts.

Here's an example of it in a chord diagram, the biggest dimension is NULL, but it's not possible to tell as NULL_STRING is omitted:

Image

For Graph, it seems transformProps returns the literal 'N/A' for null categories instead of NULL_STRING.

**3. NULL_STRING is not shown in tooltips for NULL value dimensions in certain charts**

Here's an example of a NULL value dimension for a bubble chart, the tooltip does not have NULL_STRING in the tooltip title:

Image

I have spotted this same issue in Radar charts and Treemap charts, though it could be visible in more.

**To reproduce**
1. Use a virtual dataset with NULL dimensions via SQL lab, such as:
```
SELECT 'alpha' AS category, 10 AS value
UNION ALL SELECT 'alpha', 20
UNION ALL SELECT 'alpha', 35
UNION ALL SELECT 'beta', 15
UNION ALL SELECT 'beta', 25
UNION ALL SELECT 'beta', 40
UNION ALL SELECT NULL, 12
UNION ALL SELECT NULL, 22
UNION ALL SELECT NULL, 38
```
2. Configure category as the dimension and reproduce see the different issues across the different chart types.

**Thoughts**
How null representation is defined is also scattered, refactor and consolidate a single null constant + formatter into @superset-ui/core, and pass dropna=False in postprocessing for issue 1.

### Superset version

6.1.0

### Python version

3.11

### Node version

16

### Browser

Chrome

### Additional context

I use postgres as the DB

### Checklist

- [x] I have searched Superset docs and Slack and didn't find a solution to my problem.
- [x] I have searched the GitHub issue tracker and didn't find a similar bug report.
- [x] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the NULL-dimension query across the affected chart types, then inspect aggregate.py and the chart plugins' transformProps paths, including @superset-ui/core. Compare how NULL values and NULL_STRING are handled in charts and tooltips. Done means NULL groups remain visible and NULL_STRING is displayed consistently in labels and tooltips.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python, sql
Domain
backend, data-visualization, frontend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.