GoogleCloudPlatform / GoogleCloudPlatform/bigquery-ml-utils

IMPORTANT: incorrect estimate values of logistic regression (especially when EARLY_STOP = TRUE)

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
45
Forks
16
PR merge metrics
No merged PRs in 30d

Description

I'm not sure here is the correct place to tell you this fact 🙇‍♀️
Please share this fact with the correct people if necessary.

# (1) Summary:
- estimates of logistic regression in BQML are wrong (i.e., the values are not close to the maximum likelihood estimator)
- this is because early_stop and batch_gradient_descent

# (2) Suggestion:
- set default value of EARLY_STOP FALSE
- remove the feature to compute the p-value
- implement the Newton-Raphson method (2nd conv.) and apply

(both, especially when `CALCULATE_P_VALUES = TRUE`)

I understand the above suggestion may not suit the design strategy as BQ"ML" (it's ML, but not statistics).
So, there should be better solutions.

# (3) Numerical Example:

The following queries have the same data, so the estimates should be the same.
In this example, the coefficient of x is ln3 = 1.0986...
But, coef. in query1 is 1.14 (wrong)
and coef. in query2 (no early_stop) is 1.0986... (correct!)

## data

![Screenshot 2024-06-21 at 15 37 35](https://github.com/GoogleCloudPlatform/bigquery-ml-utils/assets/8705601/2334740a-e878-4769-8609-b8fa45e02cba)

## result

result of query 1 (with early stop)
![aa](https://github.com/GoogleCloudPlatform/bigquery-ml-utils/assets/8705601/2d11c45e-1318-4445-b3d9-a90ac7760217)

result of query 2 (withOUT early stop)
![bb](https://github.com/GoogleCloudPlatform/bigquery-ml-utils/assets/8705601/a88ab2d8-533d-4d7a-888d-f8ef3953e24e)

## queries

Query1:
```create or replace
model `project.dataset.model1`
options (
input_label_cols = ['y'],
model_type = 'logistic_reg',
data_split_method = 'no_split',
max_iteration = 15,
l1_reg = 0,
l2_reg = 0,
)
as

with arrays as (
select
array[-1, -1, -1, -1, 1, 1, 1, 1] as x,
array[0, 0, 0, 1, 0, 1, 1, 1] as y
)
select
x,
y
from arrays,
unnest(x) as x with offset as index_x
join unnest(y) as y with offset as index_y
on index_x = index_y
```

Query2:

```create or replace
model `project.dataset.model2`
options (
input_label_cols = ['y'],
model_type = 'logistic_reg',
data_split_method = 'no_split',
max_iteration = 15,
l1_reg = 0,
l2_reg = 0,
optimize_strategy = 'batch_gradient_descent',
early_stop = false # DIFFERENCE HERE
)
as

with arrays as (
select
array[-1, -1, -1, -1, 1, 1, 1, 1] as x,
array[0, 0, 0, 1, 0, 1, 1, 1] as y
)
select
x,
y
from arrays,
unnest(x) as x with offset as index_x
join unnest(y) as y with offset as index_y
on index_x = index_y
```

# (4) Practical example
I can't share the original data because it is highly confidential. So, I share an abstract situation.

The numerical difference is as follows:
![Screenshot 2024-06-21 at 15 41 39](https://github.com/GoogleCloudPlatform/bigquery-ml-utils/assets/8705601/60c171c7-168f-4673-bbfa-1e61dac72246)

These are the very first coefficients of 100+ variables, 100k+ data (in my job).

When we believe the statsmodels is true, BQML cannot estimate correct estimated values.

## Additional information 1: about this model

- var_1 ~ var_3 are one-hot variables from one categorical variable.
- and, for many data, var_1 + var_2 + var_3 = 1, so there is little possibility of multi-colinearity (but max(VIF) = 5. so we can say this is not the case of multi-colinearity)

## Additional information 2: loss curve and estimation algorithm

Loss curve of this estimation is as follows.
So, as "ML," it is very natural to early_stop (because no loss improvement)
But, as "statistics," it is problematic since the estimated values are far from correct. (This, p-values are misleading)

![Screenshot 2024-06-21 at 12 43 14](https://github.com/GoogleCloudPlatform/bigquery-ml-utils/assets/8705601/e302cf0a-2cb6-45b8-beef-eb9a773c9557)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.