Breakthrough-Energy / Breakthrough-Energy/PreREISE

Bug report: EIA API for demand data is timing out

Open
#322 4 comments 0 reactions 5 assignees Claimed by @jenhagg View on GitHub
bug hifld
Dominant language
Jupyter Notebook
Stars
22
Forks
29
PR merge metrics
No merged PRs in 30d

Description

# :beetle:

- [x] I have checked that this issue has not already been reported.

### Bug summary
The EIA API is proposed to be used for demand data for the HIFLD Project as described in Issue #293
I am trying to retrieve data for the list of BAs created in issue #241
During the week of Oct 31 - Nov 4, I have been attempting to use the EIA API, with varying degrees of success due to it timing out at unpredictable intervals, while trying to retrieve data for that BA list. When it times out, it throws an error. It is also slow, taking >10 minutes per BA in some cases.

### Code for reproduction
import getpass
import pandas as pd
import from prereise.gather.demanddata.eia.get_eia_data import get_ba_demand

start = pd.to_datetime('2016-01-01 00:00:00')
end = pd.to_datetime('2016-12-31 23:00:00')

key = getpass.getpass(prompt='api_key=')
#BA shp list is list of BAs from
get_ba_demand(ba_shplist, start, end, key)`
#
#

### Actual outcome
![Screen Shot 2022-11-04 at 3 31 55 PM](https://user-images.githubusercontent.com/49412554/200084322-a2417a7b-58b3-4c92-adca-758ef294c571.png)
Other possible error I've observed: IncompleteRead: IncompleteRead(303104 bytes read)

#
#

### Expected outcome
What it looks like downloading data from the API:
![Screen Shot 2022-11-04 at 4 00 03 PM](https://user-images.githubusercontent.com/49412554/200086688-c73c413d-27a5-43c4-beb4-ea0cbd5e7595.png)

### Additional context
There are multiple warnings that may be contributing to these issues that appear on the API site as of Nov 4, 2022.
There is a 'scheduled maintenance' on Nov 4, but the issue happened other days this week also.
There is a likely more important warning "Notice: EIA will discontinue support for its legacy API (APIv1) in November, 2022. Excel add-in v1 sheets will continue to function as they are. Please refer to our [documentation](https://www.eia.gov/opendata/documentation.php) for the APIv2 interaction methods and our [APIv2 query browser](https://www.eia.gov/opendata/browser/) to view the data."
This second warning may be contributing to the issues with the API and likely requires a long term fix.
![Screen Shot 2022-11-04 at 4 04 17 PM](https://user-images.githubusercontent.com/49412554/200087048-cb7f49b7-001c-42b8-91cc-2ca0cafe99af.png)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.