Job trends in software development by term frequency

### Overview

Analyze six-month shifts in **software development job postings** by comparing **March 2025** (now a year ago) to **the most recent 30 days**, quantifying what skills/technologies are **rising**, **falling**, **newly in demand**, or **disappearing**. Outputs include frequency deltas, trend lines, and a concise narrative report that’s reproducible and portfolio-ready.

The assignee for this issue will have to wear three hats:

1) **A Data Scientist** - Responsible for posing/framing the question(s) and adapting the inquiry according to evidence discovered. An initial hypothes(e)s posed in a report from some intial motivations and investigations. Adapt that hypothesis as insights come in. Create the Final Report for peer review.
2) **A Data Engineer** - Cleaning, Prepping data after devising a strategy to scrape old posting via an API. Implement the scraping and shape the data for visualization by the Data Analyst.
3) **A Data Analyst** - Responsible for answering the Data Scientist's hypotheses, providing a report to them using the insights from the shaped data. Reports to generate along the way.

### Action Items

**The beginning (research + design)**

- [ ] * Define cohort & windows: titles containing `software`, `developer`, `engineer`, `frontend`, `backend`, `full-stack`, `mobile`, `devops`, `data engineer`, `ml engineer`; past window = **2022-11-01→2026-03-31**, current window = **last 30 days**.
- [ ] * Select sources:

- [ ] * **Federal**: USAJOBS HistoricJoa + AnnouncementText (full text, closed postings).
- [ ] * **Private sector (optional v1.1)**: archived postings from Greenhouse/Lever via Wayback snapshots.
- [ ] * Draft skills/tech lexicon with preferred mappings (e.g., `whitelist→allowlist`, `primary/replica` vs `master/slave`; frameworks: React/Vue/Svelte; platforms: Kubernetes, Terraform; languages: Rust/Go/TypeScript; AI terms: LLM, RAG, vector DB).
- [ ] * Methods spec: text extraction & normalization, n-gram and phrase detection, **log-odds with informative Dirichlet prior** for rise/fall, monthly slope estimation, multiple-testing control (BH-FDR).
- [ ] * Ethics & QA: de-duplicate pages, ignore code blocks, document gaps (unarchived pages), and record rate-limit handling.

**If researched and ready (implementation steps)**

- [ ] 1. **Ingest**

* Pull metadata with `GET /api/HistoricJoa` for both windows; join long fields with `GET /api/HistoricJoa/AnnouncementText` (key: `USAJOBSControlNumber`). ([[developer.usajobs.gov](https://developer.usajobs.gov/API-Reference/GET-api-HistoricJoa?utm_source=chatgpt.com)][1])
* (Optional) Enumerate private-sector snapshots via **Wayback CDX**; select nearest capture to target dates. ([[Internet Archive](https://archive.org/developers/wayback-cdx-server.html?utm_source=chatgpt.com)][2])
.
- [ ] 2. **Extract & Normalize** * HTML→main text via **trafilatura** (fallback: Readability/jusText), lowercase, strip boilerplate & code blocks, Unicode normalize. ([[trafilatura.readthedocs.io](https://trafilatura.readthedocs.io/?utm_source=chatgpt.com)][3])

- [ ] 3.. **Feature Build**
* Tokenize; build unigrams/bigrams; apply skill lexicon + regex for variants (e.g., `Type-Script|TypeScript`); map deprecated→preferred terms.

- [ ] 4. **Analysis**

* Compute term frequencies, ΔTF-IDF, and **log-odds** rise/fall between windows; estimate monthly slope on multi-month series if expanded.
* Tag **new** (present now, absent before) and **gone** (present before, absent now) terms; flag ambiguous terms for manual review.
- [ ] 5. **Deliverables**

* CSVs: `term_stats_past.csv`, `term_stats_current.csv`, `trend_report.csv`.
* **Dashboard** (Streamlit or Altair/Vega-Lite): top rising/falling skills, new vs gone, per-title breakdown, sample snippets.
* Methods README with caveats, rate-limit notes, and reproducibility (fixed seeds).

- [ ] 6. **Testing & Ops**

* Add unit tests for parsing & counting; retries/backoff; de-duplication via URL+digest; logs with timings and error contexts.
* Schedule optional monthly refresh via GitHub Actions (artifact publish).

### Resources/Instructions

* **USAJOBS APIs (free, historical)**

* API overview & endpoints (Search, Historic JOAs, Announcement Text). ([[developer.usajobs.gov](https://developer.usajobs.gov/api-reference/?utm_source=chatgpt.com)][4])

* **Internet Archive / Wayback (for expired private-sector postings)**

* CDX Server API reference & help pages. ([[Internet Archive](https://archive.org/developers/wayback-cdx-server.html?utm_source=chatgpt.com)][2])
* Python clients: **waybackpy** or **edgi/wayback**. ([[PyPI](https://pypi.org/project/waybackpy/?utm_source=chatgpt.com)][5])

* **Extraction**

* **trafilatura** docs & quickstart; evaluation notes. ([[trafilatura.readthedocs.io](https://trafilatura.readthedocs.io/?utm_source=chatgpt.com)][3])

* **Schema hints (when scraping non-federal sites)**

* `schema.org/JobPosting` and Google’s Job Posting structured data guide. ([[Schema.org](https://schema.org/JobPosting?utm_source=chatgpt.com)][6])

* **Suggested stack (choose 1 of each pair)**

* HTTP: `httpx` **or** `requests`
* Parsing: `trafilatura` **or** `readability-lxml`
* NLP/Features: `scikit-learn` **or** `textacy`
* Stats: `statsmodels` **or** `scipy`
* Viz/Dashboard: `Altair/Vega-Lite` **or** `Streamlit`
* Archival: `waybackpy` **or** `edgi/wayback`

* **Deliverable spec**

* Repo with `fetch/`, `extract/`, `analyze/`, `report/` modules; `config.yaml` for windows & filters; CI workflow to run end-to-end and publish CSVs + HTML report.

* If this issue requires access to 311 data, please answer:

* **Not applicable.** This project uses USAJOBS public APIs and public web archives only.

[1]: https://developer.usajobs.gov/API-Reference/GET-api-HistoricJoa?utm_source=chatgpt.com "USAJOBS - GET /api/HistoricJoa"
[2]: https://archive.org/developers/wayback-cdx-server.html?utm_source=chatgpt.com "Wayback CDX Server API - BETA"
[3]: https://trafilatura.readthedocs.io/?utm_source=chatgpt.com "A Python package & command-line tool to gather text on the ..."
[4]: https://developer.usajobs.gov/api-reference/?utm_source=chatgpt.com "USAJOBS - API Reference"
[5]: https://pypi.org/project/waybackpy/?utm_source=chatgpt.com "waybackpy"
[6]: https://schema.org/JobPosting?utm_source=chatgpt.com "JobPosting - Schema.org Type"