Stream data - refitting model results in error for 1 data point
- Dominant language
- Python
- Stars
- 20.4k
- Forks
- 4.6k
- Avg merge
- 19h 52m
- Merged PRs (30d)
- 1
Description
The use case is we're setting up prophet on IoT real-time forecasting/anomaly detection.
The plan is to "incept" the model on historical data (around the last 55_000 data points)
Serialize the model to storage.
Now run through this process on a stream of data.
1. Download model from blob.
2. Bootstrap model using refitting documentation
3. Ask for prediction for next data point ```make_future_dataframe(periods=1, freq='15Min')```
4. Is the data point coming in inside the bounds? if yes anomaly if no not anomaly.
5. Add the data point to data frame, add dataframe to existing model (using re-fitting documentation)
6. Save model back to blob.
After setting this process up we're seeing an error "Dataframe has less than 2 non-NaN rows" which upon reading through some stuff online looks like an error but the model has been trained on 55_000 data points all of which are stored inside the serailized_model.json file, could this be change to not error if the model is being refitted?
(pseudo code that causes error)
```py
df1 = pd.DataFrame([json.loads('{"ds": "2022-06-27 20:50:30", "y": "20.5"}')])
df1['ds'] = pd.to_datetime(df1['ds'])
df1 = df1.sort_values('ds')
df1 = df1.set_index('ds').resample('15Min', origin='epoch').mean().ffill().reset_index()
model = Prophet().fit(df1, init=stan_init(model_from_json(mod)))
```
Or is there another methodology you would recommend to run prophet on streaming data after an initial training session?
Contributor guide
Assessment
This issue has not been assessed yet.