QuantEcon / QuantEcon/lecture-python-intro

heavy_tails.md identifies power laws only visually — add a goodness-of-fit test

Open
#807 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jupyter Notebook
Stars
65
Forks
32
Avg merge
4d 14h
Merged PRs (30d)
6

Description

heavy_tails.md defines power laws and Pareto tails formally, then identifies them by eye: "All plots are in log-log, so that a power law shows up as a linear log-log plot, at least in the upper tail." The lecture builds empirical CCDFs and Q-Q plots for firm size and city size, and stops there.

It has no goodness-of-fit test, no tail-index estimator, and no note that log-log linearity is a weak diagnostic. Searching the lecture for "goodness", "test for", "Hill estimator", "KS test", "Kolmogorov" or "Clauset" returns nothing.

Why this matters

Exercise ht_ex4 asks the reader to compare a Pareto distribution against a mean-and-median-matched lognormal, for the present discounted value of corporate tax revenue, and to observe the difference. The lecture therefore poses the Pareto-versus-lognormal question and gives the reader no way to settle it from data. The same comparison appears, also unresolved, as Exercise 2.2.10 of Economic Networks.

Eyeballing a log-log plot for straightness is precisely the practice the goodness-of-fit literature exists to caution against, so teaching only the visual method leaves readers with a diagnostic that looks more reliable than it is.

Suggested scope

A short section, not a new lecture:

  • Estimating the tail index — the Hill estimator, and its sensitivity to where the tail is deemed to start
  • Testing the hypothesis — the Clauset–Shalizi–Newman KS-based procedure is the standard reference and has a widely used implementation
  • A sentence of honesty in the existing visual material, noting that log-log linearity is suggestive rather than conclusive

Enough that a reader can answer "is this actually a power law?" instead of "does this look straight?". Wiring it back to ht_ex4 would close the loop on an exercise that currently ends in an observation rather than an answer.

Spun out of QuantEcon/meta#141, which collected it as the one concrete deliverable inside a broader proposal.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with lectures/heavy_tails.md, especially the empirical CCDF, Q-Q plot, and ht_ex4 material. Research the Hill estimator and the Clauset–Shalizi–Newman KS-based procedure, then add a short section, qualify the existing visual diagnostic, and connect the method back to ht_ex4 so readers can assess Pareto versus lognormal data.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
data, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.