H2 — Pandemic disruption accelerated a decline that began earlier | Student decline: More than pandemic learning loss alone

# H2 — Pandemic disruption accelerated a decline that began earlier

## Research question
Did educational achievement begin deteriorating before the COVID-19 pandemic, with the pandemic producing an additional disruption rather than initiating the decline?

**Rephrased (as a proposition to investigate, not a conclusion to prove):** Does a model containing a pre-pandemic decline plus a pandemic shock explain the observed achievement trajectory better than a pandemic-only model?

**Evidence to Explore:**

* Change-point analysis identifies deterioration before 2020;
* States or districts declining before COVID continue declining afterward;
* Pre-2013 trends differ materially from 2013–2019 trends;
* Pandemic closures explain the 2020–2022 shock but not the earlier decline.

## Working Documents
- [Issue 174](https://drive.google.com/drive/u/0/folders/1F2l95aaDerpEPVvzIUXDuhDyNfgcmjk8) Google Drive Folder
- [Data Source to Hypothesis Alignment Chart](https://docs.google.com/spreadsheets/d/14ZdoOvpv6nY4ulvS1IUQmn9C2bT0H7QQSPTGqR89wE4/edit?usp=sharing) (Google Sheets)
- [SEDA AnnualSub YS data by Admin District](https://console.cloud.google.com/bigquery?ws=!1m5!1m4!4m3!1spandemic-educational-decline!2sSEDA_admindist_annualsub_ys_2025!3sSEDA_admindist_annualsub_ys_2025) Google Big Query
- [SEDA AnnualSub YS data by Admin District](https://drive.google.com/file/d/1go9Vn861l17Rf5CespF67GMo8z9SAza8/view?usp=drive_link) (csv)
- [SEDA documentation v2025.1](https://drive.google.com/file/d/1uNR4_bnsjZidFTT7Ea-1a1XWAibt0blA/view?usp=drive_link) PDF
- [SEDA Codebook (Admin Dist)](https://docs.google.com/spreadsheets/d/1ybdo1z9d4FAwNHDNHrw7vMuy73Awqs44/edit?usp=drive_link&ouid=115952167676069022141&rtpof=true&sd=true) Google Sheet

## Resources
- [NAEP Data Service API](https://www.nationsreportcard.gov/api_documentation.aspx)
- [Education Scorecard](https://educationscorecard.org/)
- Stanford Education Data Archive (SEDA) [Data Downloads](https://edopportunity.org/opportunity/data/downloads/) | [Explorer](https://edopportunity.org/trends/explorer/)
- Urban Institute Education Data Portal [Main Portal](https://educationdata.urban.org/documentation/) | [School Districts](https://educationdata.urban.org/documentation/school-districts.html)
- EdFacts [Initiative](https://www.ed.gov/data/edfacts-initiative) | [Report cards](https://www.ed.gov/birth-to-grade-12-education/elementary-and-secondary-education/where-can-i-find-my-state-report-card-website)
- [Common Core of Data (CCD)](https://nces.ed.gov/ccd/files.asp)

## Research Plan

Step 1 — Translate the broad question into answerable subquestions

### Step 1 — Translate the broad question into answerable subquestions
Before choosing a statistical method, identify what would need to be true for H2 to be supported.

For example:

1. **Was achievement already declining before 2020?**
2. **Is there evidence that the pre-pandemic trajectory changed around 2013?**
3. **Was the 2020–2022 change substantially different from the pre-pandemic trend?**
4. **Did places declining before the pandemic tend to remain worse off afterward?**
5. **Does a model containing both a pre-pandemic decline and a pandemic shock explain the data better than a model attributing the decline only to COVID?**

Some potential mathematical trends to search for:
1. What were the rolling rates of change in achievement pre- and peri-pandemic?
2. Are any critical points in the rates of change statistically significant?

Step 2 — Define exactly what will count as evidence

### Step 2 — Define exactly what will count as evidence
Decide which concepts in the research question need measurable definitions.

For this hypothesis, the major concepts are:

**Achievement**

- Reading achievement;
- Mathematics achievement;
- potentially grade-specific achievement;
- potentially subgroup or distributional measures if available.

**Pre-pandemic decline**

- A negative achievement trend before 2020;
- potentially a change in slope between an earlier period and approximately 2013–2019.

**Pandemic disruption**

- A departure from the expected pre-pandemic trajectory around 2020–2022.

**Recovery**

- The trajectory after testing resumes;
- whether districts return toward their previous trajectory, remain below it, or continue deteriorating.

At this stage, don't require every possible outcome. Start with the variables actually needed to answer H2.

Step 3 — Understand and document the data

### Step 3 — Understand and document the data

This should happen **before statistical modeling**.

Create a data dictionary for each primary dataset containing at minimum:

| Field | What to document |
| --- | --- |
| Variable name | Name in source dataset |
| Meaning | What the variable measures |
| Unit | Scale, percentage, standardized score, etc. |
| Geography | State, district, school |
| Time | Available years |
| Subject | Reading/math/etc. |
| Grade | Available grade levels |
| Population | Students represented |
| Missingness | When/why observations may be absent |
| Comparability | Whether values can safely be compared across years |
| Source | SEDA, Education Scorecard, etc. |

Also document the **unit of observation**.

For example:

>
> One row = district × year × subject × grade

or whatever the source actually provides.

Then answer basic feasibility questions:

- Which years overlap?
- Is 2013 directly observed?
- Is 2019 directly observed?
- How are 2020 and 2021 represented?
- When does post-pandemic testing resume?
- Are scores on a common scale?
- Are district identifiers stable?
- Are there districts entering/leaving the dataset?
- Are there changes in assessment methodology?

This step prevents a huge amount of bad analysis later.

Step 4 — Develop testable statistical hypotheses

### Step 4 — Develop testable statistical hypotheses

Rather than treating the original H2 statement as one giant hypothesis, break it into several tests.

#### Test A — Was there a pre-pandemic decline?

**Null hypothesis H0A​:**

>
> Achievement did not decline during the pre-pandemic period.

Conceptually:
βpre-COVID​≥0
**Alternative HAA​:**

>
> Achievement declined during the pre-pandemic period.

βpre-COVID​<0
* * *

#### Test B — Did the trajectory change before COVID?

This addresses the proposed 2013 change point.

**H0B​:**

>
> The achievement trend after approximately 2013 was not meaningfully different from the earlier trend.

β2003−2012​=β2013−2019​
**HAB​:**

>
> The 2013–2019 trend was more negative than the earlier trend.

β2013−2019​<β2003−2012​
The exact breakpoint should ultimately be tested rather than assumed.

* * *

#### Test C — Was there an additional pandemic shock?

**H0C​:**

>
> Achievement after the pandemic remained consistent with the pre-pandemic trajectory.

**HAC​:**

>
> Achievement after the pandemic was significantly below what would have been expected from the pre-pandemic trajectory.

This is where an interrupted-time-series or segmented-regression framework becomes useful.

* * *

#### Test D — Were pre-pandemic declines associated with post-pandemic outcomes?

At the district level:

**H0D​:**

>
> A district's pre-pandemic achievement trend has no relationship with its pandemic or post-pandemic performance.

**HAD​:**

>
> Districts experiencing greater pre-pandemic declines tend to experience poorer post-pandemic achievement or recovery.

That doesn't establish causation, but it directly tests the “accelerated rather than created” idea.

Step 5 — Conduct exploratory data analysis

### Step 5 — Conduct exploratory data analysis

Before fitting formal models, learn what the data look like.

A useful first-pass sequence would be:

#### A. Coverage

Count observations by:

- Year;
- state;
- district;
- subject;
- grade.

Look for gaps and changes in coverage.

#### B. Trend plots

Plot:

- National/pooled reading achievement over time;
- National/pooled mathematics achievement over time;
- Individual state trajectories;
- representative district trajectories.

Mark important periods visually:

- earlier baseline;
- proposed pre-pandemic decline;
- pandemic disruption;
- recovery.

#### C. Distribution of district trends

Calculate preliminary district-level slopes and examine their distribution.

For example:
Slope2013−2019​
This helps answer:

>
> Was the apparent decline widespread, or driven by a relatively small number of districts?

#### D. Missingness and unusual observations

Investigate:

- Missing years;
- sudden district jumps;
- district consolidations;
- changes in sample composition;
- unusually large year-to-year changes.

EDA should be treated as **question-generating**, not hypothesis confirmation.

Step 6 — Choose analysis methods based on what the data support

### Step 6 — Choose analysis methods based on what the data support

Only after Steps 1–5 should the researcher settle on the formal methodology.

For H2, a sensible progression might be:

#### Analysis 1 — Simple period trends

Estimate separate trends for:
Early period2013–2019Post-pandemic
This provides an understandable baseline.

* * *

#### Analysis 2 — Segmented regression

Fit a model allowing the slope to change over time.

Conceptually:
Yt​=β0​+β1​Time+β2​PostBreak+β3​TimeAfterBreak+ϵt​
Potential breakpoints could include:

- a data-derived pre-pandemic breakpoint;
- the pandemic disruption.

This lets you distinguish:

**underlying trend**

from

**level shock**

from

**change in trajectory**.

* * *

#### Analysis 3 — Change-point detection

Rather than simply declaring 2013 the start of the decline, ask:

>
> Where does the data indicate that the trajectory changed?

If an estimated breakpoint occurs around 2013, that strengthens the original proposition.

If it occurs in 2015, 2011, or nowhere at all, that is scientifically important too.

* * *

#### Analysis 4 — District-level persistence

Estimate each district's pre-pandemic trajectory and compare it with later outcomes.

For example:
PostPandemicOutcomei​=α+βPrePandemicSlopei​+ϵi​
Later models could introduce relevant controls if justified.

Step 7 — Compare competing explanations

### Step 7 — Compare competing explanations
This is one of the most important parts of H2.

Instead of merely asking whether COVID hurt achievement, compare models representing different explanations.

#### Model 1 — Pandemic-only explanation

Achievement is relatively stable beforehand, followed by a COVID-era disruption.

#### Model 2 — Pre-existing decline

Achievement begins declining before COVID, without a distinct additional pandemic effect.

#### Model 3 — Pre-existing decline + pandemic shock

Achievement begins declining earlier **and** experiences an additional pandemic-era disruption.

Then ask:

>
> Which explanation fits the observed data best?

That gets much closer to the scientific question than simply testing whether 2022 scores were below 2019 scores.

Step 8 — Perform targeted robustness checks

### Step 8 — Perform targeted robustness checks

Robustness checks should follow the core analysis rather than becoming another enormous research project.

Repeat the central tests where practical:

- Reading vs. mathematics;
- Different grade levels;
- States vs. districts;
- Alternative pre-pandemic starting dates;
- Alternative breakpoint specifications;
- Balanced panel only vs. all available districts;
- SEDA vs. Education Scorecard estimates;
- NAEP as an independent national/state comparison.

If the conclusion disappears whenever one modeling choice changes, that is important evidence about its uncertainty.

Step 9 — Interpret the results against the original research question

### Step 9 — Interpret the results against the original research question

At the end, return explicitly to H2.

Possible conclusions might be:

#### Supported

Evidence indicates a meaningful pre-pandemic deterioration, followed by an additional pandemic shock.

#### Partially supported

There is evidence of pre-pandemic deterioration, but the pandemic effect or persistence relationship is weak or inconsistent.

#### Not supported

Achievement appears broadly stable before COVID, with the primary deterioration occurring during the pandemic.

#### Inconclusive

Data coverage, measurement changes, uncertainty, or conflicting sources prevent distinguishing the explanations.

That last category matters. A research issue should not force every analysis into “confirmed/rejected.”

Step 10 — Document what was learned and what should happen next

### Step 10 — Document what was learned and what should happen next

The subissue should end with a concise record of:

- Datasets used;
- variables used;
- exclusions;
- data dictionary;
- analysis code/notebooks;
- assumptions;
- statistical results;
- visualizations;
- limitations;
- conclusion;
- unanswered questions;
- recommended follow-up analysis.

This creates a reproducible research trail rather than just a collection of charts.

## Overall Plan:
**1. Clarify the research question**
What is the broad proposition actually asking?

**2. Operationalize the concepts**
What observable quantities represent those ideas?

**3. Understand the available data**
Build the data dictionary, establish units of observation, coverage, quality, and limitations.

**4. Form testable hypotheses**
Convert the broad research question into specific H0​/HA​ pairs.

**5. Explore the data**
Describe distributions, trends, missingness, anomalies, and relationships without prematurely claiming conclusions.

**6. Select appropriate analytical methods**
Choose methods based on the question and the data—not because they appeared in the original issue.

**7. Test competing explanations**
Where possible, compare alternative models rather than looking only for evidence supporting the proposed explanation.

**8. Check robustness**
Determine whether the result survives reasonable alternative specifications and datasets.

**9. Draw a bounded conclusion**
Supported, partially supported, unsupported, or inconclusive—with uncertainty and limitations clearly stated.

**10. Preserve reproducibility**
Document data, transformations, code, decisions, results, and next questions.