llnl / llnl/thicket

stats: std population or sample calculation on 1 value

Open
#231 0 comments 0 reactions 0 assignees View on GitHub
area-stats
Dominant language
JavaScript
Stars
18
Forks
10
PR merge metrics
No merged PRs in 30d

Description

Calling `th.stats.std(ttk, cols)` may result in aggregation of a single row. In thicket, we are calling `.agg(np.std)`, which calculates the standard deviation with a degrees of freedom (`ddof`) of 1. In other words, it divides by `n-1` (where n is the number of elements). This is statistically appropriate for estimating the population standard deviation from a sample. However, with only one element, the calculation becomes 0/0, resulting in a NaN value.

One alternative is to set `ddof=0`, calculating the standard deviation with ddof=0, dividing by n instead of n-1, resulting in a standard deviation of 0 for a single element:
```
import pandas as pd
import numpy as np

df = pd.DataFrame({'A': [1]})

# Calculate standard deviation with ddof=0
result = df.agg(lambda x: np.std(x, ddof=0))
print(result)
```

For standard deviation, it may be appropriate to have an option to toggle between population and sample calculation.

Contributor guide

Open the contributing guide

Research direction

Start at the th.stats.std(ttk, cols) entry point and trace the .agg(np.std) call described in the issue. Confirm the current single-row result, then determine how the population-versus-sample choice should be represented; the work is done when the selected behavior is defined and the one-value aggregation no longer produces an unintended result.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.