hackforla / hackforla/data-science

The challenge of powering national growth with today's data center expansion

Open
#276 0 comments 0 reactions 0 assignees View on GitHub
complexity: Large CoP: Data Science project duration: one time role: data science size: 8pt
Dominant language
Jupyter Notebook
Stars
33
Forks
22
PR merge metrics
No merged PRs in 30d

Description

# Investigate Whether the U.S. Electric Grid Can Support Future Economic Growth

## Overview

After nearly two decades of mostly flat electricity consumption, U.S. power demand is rising again. Data centers, artificial intelligence, advanced manufacturing, building electrification, electric vehicles, and population growth are creating large new loads that can appear faster than utilities can build generation and transmission.

Between 2005 and 2019, U.S. electricity demand grew by approximately 0.1% annually. Between 2020 and 2025, it grew by about 1.7% annually, with data centers becoming a major contributor. The Energy Information Administration expects the strongest four-year growth in U.S. electricity demand since 2000.

The physical grid is not expanding at the same speed. A large data center may be built in two or three years, while major transmission upgrades can require seven years or longer. The Department of Energy’s 2026 National Transmission Needs Study identifies substantial present and future transmission constraints caused by data centers, manufacturing, industrial loads, electrification, and extreme-weather risks.

At the end of 2025, approximately 8,200 generation and storage projects were seeking transmission interconnection, representing roughly 1,312 gigawatts of generation and 749 gigawatts of storage. Most proposed projects will not be completed, and projects that do reach operation are taking longer to move through interconnection studies.

The central research question is:

> **Can the U.S. electric grid add and deliver electricity quickly enough to support economic growth while maintaining reliability, affordability, and efficient use of existing resources?**

The investigation should determine whether the greatest constraints are:

* Insufficient electricity generation;
* Insufficient transmission capacity;
* Slow generator or large-load interconnection;
* Inaccurate demand forecasts;
* Local distribution-system limits;
* Inadequate storage or flexible demand;
* Poor regional coordination;
* Market and regulatory structures;
* Extreme-weather vulnerability;
* Improper allocation of infrastructure costs.

The recommended primary units of analysis are:

* Balancing-authority-hour;
* Utility- or service-territory-year;
* ISO/RTO zone-hour;
* Interconnection-project;
* Transmission-project;
* Data-center or large-load connection request.

---

## Action Items

### Initial Evidence

Several trends justify the investigation:

* U.S. electricity demand has shifted from long-term stagnation to sustained growth, with large computing facilities playing an important role.
* EIA projects that electricity used by data-center servers could reach between 446 and 818 billion kilowatt-hours by 2050, depending on growth assumptions.
* DOE reports a pressing need for additional transmission because of data centers, domestic manufacturing, industrial growth, electrification, and severe-weather exposure.
* FERC ordered all six federally regulated regional grid operators in June 2026 to justify or reform the rules governing connections for data centers and other large loads.
* Berkeley Lab identified more than 40 potential reforms for accelerating large-load connections across forecasting, interconnection, procurement, markets, operations, ratemaking, and cost allocation.
* In California, renewable generation is increasingly curtailed because generation, storage, transmission, and demand are not always geographically or temporally aligned. In the first half of 2026, reported curtailment exceeded the total for all of 2025.
* Utilities risk overbuilding generation and network infrastructure if speculative data-center requests are treated as firm demand. Consumers may then be left paying for facilities that are underused or never needed.
* Extreme heat and other weather events increase electricity demand while also reducing the performance of power plants and transmission systems.

These facts demonstrate increasing pressure but do not establish that a national capacity shortage is inevitable. Some constraints may be solved more cheaply by using existing infrastructure more efficiently.

### Hypotheses

#### H1 — Transmission is a larger constraint than generation capacity

The United States may possess or have proposed sufficient generating capacity but lack the transmission needed to move electricity from available resources to growing demand centers.

**Evidence to seek:**

* Congestion costs rise faster than total electricity demand;
* Regions with transmission expansion experience fewer price separations;
* Generation projects withdraw because of network-upgrade costs;
* Curtailment is concentrated behind transmission constraints;
* New transmission produces measurable reliability and price benefits.

#### H2 — Interconnection procedures are delaying otherwise viable generation and storage

Sequential studies, uncertain upgrade costs, speculative applications, and repeated restudies may prevent viable projects from reaching operation.

**Evidence to seek:**

* Queue duration has increased;
* Withdrawal rates vary systematically by region and technology;
* Projects face large and unpredictable network-upgrade costs;
* Cluster-study or readiness reforms improve completion rates;
* Financial deposits reduce speculative requests without excluding viable projects.

#### H3 — Large-load forecasting substantially overstates or understates future demand

Data-center developers may submit duplicate requests in multiple territories, while utilities may also fail to anticipate genuine load growth.

**Evidence to seek:**

* Requested load greatly exceeds completed load;
* The same projects appear in multiple connection queues;
* Forecast error rises in high-data-center regions;
* Probabilistic forecasts outperform single deterministic projections;
* Utilities using milestone-based forecasting make fewer stranded investments.

#### H4 — Data centers can become flexible grid resources rather than inflexible loads

Some computing tasks can be shifted by time, location, or workload without materially reducing service quality.

**Evidence to seek:**

* Flexible data-center demand reduces peak load;
* Workload shifting lowers congestion and marginal emissions;
* Onsite batteries provide grid services;
* Interruptible tariffs reduce required generation reserves;
* Flexibility contracts accelerate connections without reducing reliability.

#### H5 — Storage and improved regional markets can reduce curtailment more cheaply than additional generation

Periods of excess electricity may coexist with shortages at other hours or locations.

**Evidence to seek:**

* Storage reduces renewable curtailment and peak prices;
* Larger regional markets increase use of available generation;
* Transmission and storage are complementary rather than interchangeable;
* Marginal storage value depends strongly on duration and location;
* Demand shifting produces benefits similar to short-duration storage.

#### H6 — Poor cost allocation transfers large-load infrastructure risk to ordinary customers

Utilities may build generation, transmission, or distribution assets for speculative large customers and recover the costs from all ratepayers.

**Evidence to seek:**

* Residential rates rise after large-load infrastructure approvals;
* Data-center tariffs fail to recover stranded costs;
* Minimum-demand and exit-fee provisions reduce consumer exposure;
* Infrastructure costs differ according to regulatory design;
* Large-load customers receive subsidies not justified by broader economic benefits.

#### H7 — Retiring firm generation before replacement capacity is deliverable increases reliability risk

The issue may be one of timing rather than technology.

**Evidence to seek:**

* Reserve margins deteriorate when retirements precede new connections;
* Regions with delayed replacements experience more scarcity pricing;
* Capacity accreditation fails to reflect extreme-weather performance;
* Resource portfolios with diverse generation and storage outperform narrower portfolios;
* Planned retirements are inconsistent with realistic project-completion probabilities.

#### H8 — Extreme weather is exposing regional dependence on correlated resources

Heat, drought, wildfire, freezing temperatures, storms, and fuel-supply failures can simultaneously affect multiple assets.

**Evidence to seek:**

* Extreme-weather events produce correlated generation and transmission outages;
* Reserve calculations underestimate weather-dependent failures;
* Interregional transfer capacity improves resilience;
* Weatherization investments reduce outage duration;
* Distributed energy resources improve selected local outcomes.

#### H9 — Local distribution systems may be a hidden bottleneck

Even where wholesale generation and transmission are adequate, substations, transformers, feeders, and distribution equipment may delay new loads.

**Evidence to seek:**

* Distribution upgrades account for substantial connection delays;
* Transformer shortages affect project timelines;
* Hosting-capacity limits predict delayed electrification;
* Local grid modernization reduces connection cost and duration;
* Utility-level data reveal constraints hidden in national statistics.

#### H10 — The U.S. grid problem differs substantially by region

Some regions may primarily need transmission, others firm capacity, storage, distribution upgrades, market reform, or demand flexibility.

**Evidence to seek:**

* Stable regional constraint clusters emerge;
* National averages obscure different hourly and seasonal problems;
* The same intervention has different marginal value across regions;
* Regional coordination reduces total infrastructure requirements;
* Local resource portfolios explain differences in reliability and cost.

---

### Research Plan

#### 1. Define the principal outcomes

Measure:

* Electricity demand growth;
* Peak demand;
* Reserve margin;
* Unserved energy;
* Outage frequency and duration;
* Congestion costs;
* Wholesale prices;
* Retail rates;
* Renewable curtailment;
* Interconnection duration;
* Project completion and withdrawal;
* Transmission-project duration;
* Forecast error;
* Carbon intensity;
* Infrastructure cost per added megawatt.

Avoid classifying a grid as successful merely because it avoids blackouts. Reliability, cost, delivery speed, and resource efficiency must all be considered.

#### 2. Build an hourly regional grid dataset

Collect by balancing authority or ISO/RTO zone:

* Load;
* Generation by fuel;
* Imports and exports;
* Wholesale prices;
* Operating reserves;
* Storage charging and discharge;
* Curtailment;
* Weather;
* outages;
* congestion;
* emissions.

Calculate:

Image

Image

Image

#### 3. Measure demand growth and forecast accuracy

Compare utility and regional forecasts with actual load.

For each forecast:

Image

Calculate:

* Mean absolute error;
* Percentage error;
* Peak-load error;
* Error by forecast horizon;
* Error in regions with large data-center queues.

Develop low, central, and high-demand scenarios rather than one forecast.

#### 4. Build a large-load project dataset

For each data center, factory, hydrogen facility, or other major load, collect:

* Requested capacity;
* Location;
* Request date;
* Developer;
* Expected operating date;
* Project status;
* Utility territory;
* Required upgrades;
* Onsite generation;
* Storage;
* Flexible-load capability;
* Public incentives;
* Water demand where relevant.

Deduplicate projects appearing in multiple public filings or connection requests.

#### 5. Analyze generator interconnection queues

Use Berkeley Lab’s project-level queue dataset to examine:

* Technology;
* capacity;
* request date;
* proposed operating date;
* withdrawal;
* completion;
* region;
* developer;
* network-upgrade cost where available.

Estimate project survival:

Image

Use survival analysis to estimate time from request to operation and identify factors associated with withdrawal.

#### 6. Measure transmission constraints

Construct regional measures from:

* Locational marginal-price differences;
* Congestion revenue;
* transmission outages;
* transfer limits;
* curtailment;
* interregional flows;
* transmission additions.

Evaluate major transmission projects using event studies:

Image

Possible outcomes include:

* Congestion;
* price volatility;
* curtailment;
* outage risk;
* generation investment;
* retail cost.

#### 7. Model resource adequacy under uncertainty

Develop scenarios combining:

* Demand growth;
* weather;
* plant retirements;
* generation completion probabilities;
* storage duration;
* fuel availability;
* transmission limits;
* demand flexibility.

Use Monte Carlo simulation rather than treating every proposed project as certain.

Estimate:

* Loss-of-load probability;
* Expected unserved energy;
* reserve shortfall;
* cost;
* emissions.

#### 8. Evaluate flexible-demand scenarios

Model large loads as:

* Constant;
* Curtailable;
* Shiftable within hours;
* Shiftable across regions;
* Supported by onsite batteries;
* Supported by onsite generation.

Compare effects on:

* Peak load;
* system cost;
* congestion;
* emissions;
* connection timing;
* reliability.

#### 9. Analyze consumer cost allocation

For selected utility territories, collect:

* Approved capital projects;
* rate cases;
* large-load tariffs;
* customer contributions;
* minimum bills;
* contract terms;
* exit fees;
* stranded-cost provisions.

Estimate how much infrastructure cost is paid by:

* The large-load customer;
* Existing residential customers;
* Commercial customers;
* Taxpayers;
* Future ratepayers.

#### 10. Study extreme-weather resilience

Link hourly weather with:

* Demand;
* plant deratings;
* fuel shortages;
* transmission outages;
* wholesale prices;
* customer outages.

Model correlated failures rather than independent component outages.

Potential events include:

* Heat waves;
* winter storms;
* drought;
* wildfire;
* hurricanes;
* flooding.

#### 11. Identify regional grid archetypes

Cluster regions using:

* Demand growth;
* reserve margin;
* interconnection delay;
* congestion;
* curtailment;
* weather exposure;
* resource mix;
* storage;
* transmission capacity;
* retail-price growth.

Possible classifications:

* High-growth capacity-constrained;
* Transmission-constrained renewable regions;
* Firm-capacity retirement risk;
* Distribution-constrained urban areas;
* High-curtailment systems;
* Flexible, well-integrated systems.

#### 12. Perform robustness checks

Repeat analysis using:

* Alternative demand scenarios;
* Different weather years;
* Proposed versus probability-adjusted projects;
* Different storage durations;
* Flexible and inflexible loads;
* Alternative outage-cost assumptions;
* Regional and national transmission expansion;
* With and without speculative load requests;
* Multiple carbon-price assumptions, including zero.

#### 13. Produce the Final PowerPoint

Recommended structure:

1. U.S. electricity demand is growing again;
2. What is driving the increase;
3. How the grid delivers electricity;
4. Generation versus transmission constraints;
5. Interconnection queues;
6. Data-center and manufacturing loads;
7. Forecast uncertainty;
8. Storage, regional markets, and flexible demand;
9. Extreme-weather reliability;
10. Who pays for grid expansion;
11. Regional findings;
12. Highest-value interventions.

The presentation must distinguish:

* Energy from capacity;
* Average load from peak load;
* Proposed projects from completed projects;
* Generation capacity from deliverable capacity;
* Transmission from distribution constraints;
* Reliability from affordability;
* Connection requests from credible demand;
* Curtailment from lack of generation.

---

## Resources

### Data, APIs, and Where to Get Data

#### U.S. Energy Information Administration API

The EIA API provides electricity data covering:

* Hourly demand;
* generation;
* interchange;
* retail sales;
* prices;
* fuel use;
* generation capacity;
* power plants;
* balancing authorities.

Use for:

* National and regional load growth;
* hourly generation mix;
* power-sector forecasts;
* retail rates;
* plant additions and retirements.

EIA’s 2026 outlook confirms that computing facilities are a major source of renewed electricity-demand growth.

#### Berkeley Lab Queued Up Dataset

Berkeley Lab publishes a free project-level Excel dataset covering all seven ISO/RTO regions and 50 non-ISO utilities, representing approximately 98% of installed U.S. generating capacity.

Use for:

* Generator and storage interconnection requests;
* Queue duration;
* project status;
* capacity;
* technology;
* region;
* completion and withdrawal analysis.

This dataset does not include every large-load connection request.

#### FERC eLibrary and Data

Use for:

* Regional transmission filings;
* rate cases;
* tariffs;
* interconnection reforms;
* transmission planning;
* large-load proceedings;
* reliability filings.

FERC’s June 2026 large-load orders provide a current policy framework for comparing regional connection rules.

#### DOE National Transmission Needs Study

Use for:

* Present and projected transmission constraints;
* congestion;
* transfer needs;
* regional planning needs;
* load-growth scenarios.

The draft 2026 study identifies transmission needs related to computing, manufacturing, electrification, growth, and severe weather.

#### NERC Reliability Assessments

Use:

* Long-Term Reliability Assessment;
* Seasonal Reliability Assessments;
* State of Reliability;
* Event analyses.

Variables include:

* Peak demand;
* reserve margins;
* generation availability;
* risk areas;
* planned additions and retirements.

The 2025 Long-Term Reliability Assessment was presented by FERC and NERC in February 2026.

#### ISO and RTO Public APIs and Downloads

Potential systems include:

* CAISO;
* ERCOT;
* PJM;
* MISO;
* Southwest Power Pool;
* New York ISO;
* ISO New England.

Use for:

* Locational marginal prices;
* load;
* generation;
* congestion;
* reserves;
* outages;
* curtailment;
* interconnection queues;
* transmission projects;
* market settlements.

Data formats differ substantially and will require regional adapters.

#### EPA eGRID

Use for:

* Power-plant emissions;
* generation;
* fuel;
* grid-region carbon intensity;
* plant location.

#### EPA CAMPD API

The Clean Air Markets Program Data API provides hourly emissions and operating data for many fossil-fuel power units.

Use for:

* Hourly generation proxies;
* carbon dioxide;
* sulfur dioxide;
* nitrogen oxides;
* unit operations;
* plant outages or deratings where inferable.

#### NOAA and National Weather Service APIs

Use for:

* Temperature;
* heat waves;
* storms;
* wind;
* solar conditions;
* drought;
* hurricanes;
* extreme-weather events.

#### Department of Energy Open Data

Use for:

* Transmission;
* storage;
* resilience;
* energy infrastructure;
* federal investments;
* technology demonstration projects.

#### USAspending API

Use for:

* Grid-modernization grants;
* transmission investments;
* storage demonstrations;
* resilience projects;
* manufacturing awards;
* federal energy contracts.

#### Form EIA-860 and EIA-923

Use for:

* Generator characteristics;
* planned additions;
* retirements;
* fuel;
* generation;
* capacity;
* operating status.

#### EIA-861 and EIA-861M

Use for:

* Utility sales;
* customer counts;
* demand;
* distributed generation;
* demand response;
* utility characteristics.

#### Utility Integrated Resource Plans

Use for:

* Demand forecasts;
* planned generation;
* retirements;
* data-center assumptions;
* transmission needs;
* cost projections.

These often require manual extraction from state regulatory filings.

#### State Public Utility Commission Records

Use for:

* Rate cases;
* certificates of need;
* large-load tariffs;
* data-center contracts;
* cost allocation;
* consumer protections;
* resource approvals.

#### Homeland Infrastructure Foundation-Level Data

Where currently available, use geospatial layers for:

* Transmission lines;
* substations;
* power plants;
* energy infrastructure.

Security-sensitive attributes must not be reconstructed or published at unsafe levels of detail.

#### OpenFEMA

Use for:

* Disaster declarations;
* hazard exposure;
* infrastructure assistance;
* resilience spending.

#### Lawrence Berkeley National Laboratory Large-Load Research

The 2026 *Speed to Power* report organizes more than 40 possible solutions into forecasting, interconnection, procurement, markets, operations, ratemaking, and cost allocation.

### Foundational Resources

* EIA, *Strongest Four-Year Growth in U.S. Electricity Demand Since 2000*.
* EIA, *Fossil Generation Could Rise with Faster-than-Expected Data-Center Demand*.
* DOE, *2026 Draft National Transmission Needs Study*.
* Berkeley Lab, *Queued Up: 2026 Edition*.
* Berkeley Lab, *Speed to Power*.
* National Academies, *As Electricity Demand Grows and Risks Increase, Experts Examine How the Grid Can Keep Up*.
* Reuters, *Rising Curtailments in California Underline U.S. Grid Ordeal*.

### Central Scientific Caution

Interconnection queues are not forecasts. They contain speculative, duplicated, immature, and ultimately withdrawn projects. Likewise, announced data-center demand is not necessarily load that will be constructed.

The strongest investigation will compare:

1. Requested capacity;
2. Contracted capacity;
3. Capacity under construction;
4. Completed capacity;
5. Actual hourly demand;
6. Available generation;
7. Deliverable generation;
8. Transmission and distribution limits;
9. Weather-adjusted reliability;
10. Consumer cost.

The most valuable result may show that the grid does not face one national electricity shortage. It faces a collection of **regional timing, location, forecasting, coordination, and cost-allocation problems** requiring different interventions.

- If this issue requires access to 311 data, please answer the following questions:
- Do you need a one-time or ongoing dump of the data?
- Do you need subset of data (i.e. certain years) or the entire data set (approx. 4 million rows or 11 GB)?
- If a subset is needed, please define subset characteristics (i.e. date range, etc.)
- Do you need online access via an API or a download of data?

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or executable entry point are identified in the issue. Start by reviewing the repository structure and narrowing the research question, units of analysis, data sources, and outcomes; done should include a defined analysis with evidence addressing the selected constraints.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.