GoogleCloudPlatform / GoogleCloudPlatform/bigquery-ml-utils
IMPORTANT: incorrect estimate values of logistic regression (especially when EARLY_STOP = TRUE)
- Dominant language
- Jupyter Notebook
- Stars
- 45
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
I'm not sure here is the correct place to tell you this fact 🙇♀️
Please share this fact with the correct people if necessary.
# (1) Summary:
- estimates of logistic regression in BQML are wrong (i.e., the values are not close to the maximum likelihood estimator)
- this is because early_stop and batch_gradient_descent
# (2) Suggestion:
- set default value of EARLY_STOP FALSE
- remove the feature to compute the p-value
- implement the Newton-Raphson method (2nd conv.) and apply
(both, especially when `CALCULATE_P_VALUES = TRUE`)
I understand the above suggestion may not suit the design strategy as BQ"ML" (it's ML, but not statistics).
So, there should be better solutions.
# (3) Numerical Example:
The following queries have the same data, so the estimates should be the same.
In this example, the coefficient of x is ln3 = 1.0986...
But, coef. in query1 is 1.14 (wrong)
and coef. in query2 (no early_stop) is 1.0986... (correct!)
## data

## result
result of query 1 (with early stop)

result of query 2 (withOUT early stop)

## queries
Query1:
```create or replace
model `project.dataset.model1`
options (
input_label_cols = ['y'],
model_type = 'logistic_reg',
data_split_method = 'no_split',
max_iteration = 15,
l1_reg = 0,
l2_reg = 0,
)
as
with arrays as (
select
array[-1, -1, -1, -1, 1, 1, 1, 1] as x,
array[0, 0, 0, 1, 0, 1, 1, 1] as y
)
select
x,
y
from arrays,
unnest(x) as x with offset as index_x
join unnest(y) as y with offset as index_y
on index_x = index_y
```
Query2:
```create or replace
model `project.dataset.model2`
options (
input_label_cols = ['y'],
model_type = 'logistic_reg',
data_split_method = 'no_split',
max_iteration = 15,
l1_reg = 0,
l2_reg = 0,
optimize_strategy = 'batch_gradient_descent',
early_stop = false # DIFFERENCE HERE
)
as
with arrays as (
select
array[-1, -1, -1, -1, 1, 1, 1, 1] as x,
array[0, 0, 0, 1, 0, 1, 1, 1] as y
)
select
x,
y
from arrays,
unnest(x) as x with offset as index_x
join unnest(y) as y with offset as index_y
on index_x = index_y
```
# (4) Practical example
I can't share the original data because it is highly confidential. So, I share an abstract situation.
The numerical difference is as follows:

These are the very first coefficients of 100+ variables, 100k+ data (in my job).
When we believe the statsmodels is true, BQML cannot estimate correct estimated values.
## Additional information 1: about this model
- var_1 ~ var_3 are one-hot variables from one categorical variable.
- and, for many data, var_1 + var_2 + var_3 = 1, so there is little possibility of multi-colinearity (but max(VIF) = 5. so we can say this is not the case of multi-colinearity)
## Additional information 2: loss curve and estimation algorithm
Loss curve of this estimation is as follows.
So, as "ML," it is very natural to early_stop (because no loss improvement)
But, as "statistics," it is problematic since the estimated values are far from correct. (This, p-values are misleading)

Contributor guide
Assessment
This issue has not been assessed yet.