A method that claims to be right 95% of the time is making a statement anyone can count. Almost nobody counts it.
The interval taught first in every introductory course covers 87.6% of the time when it says 95%, and the failure is worst exactly where proportions are usually reported — a rare event in a small sample. That is not an estimate. For a proportion the sample space is finite, so the coverage is a sum over every outcome that could have occurred, and the number is exact. The same discipline applies to everything here: a test's p-values are checked for flatness, a simulation carries its seed, and every claim that a picture makes has been run across many seeds rather than one.
Start anywhere
twelve of 436, in 93 fields
Every essay sits in one of 93 fields, grouped into 8 parts of the subject, on a ladder of arguments against one idea, and under the threads below. The same material is also indexed by the generator that drew each figure and by every object more than one essay names, and search runs over all of it in the browser.Coming back? What's new lists the essays in the order they arrived rather than the order they are meant to be read.
The distribution itself
Sums of almost anything
The theorem says sums converge on one shape whatever they are sums of, which is remarkable and true. Watching it happen from a one-sided skewed source, with the rate of convergence predicted in advance, is more convincing than watching the shape appear.
The distribution itself
The shape, and where its mass is
68, 95, 99.7 is recited more often than any other set of numbers in the subject. They are integrals of a specific curve, they are worth computing rather than remembering, and the third one is the one people misuse.
The distribution itself
What normal actually looks like
A single quantile plot of forty normal points wanders enough to look suspicious. Twenty of them, all genuinely normal, show what the noise looks like — and any single panel a reader would have rejected is in there.
Intervals, counted
What the 95% refers to
An interval that claims 95% is making a checkable statement about a procedure, not about the interval in front of you. Build every possible sample and count, and the interval taught first turns out to cover 87.6% of the time.
Intervals, counted
More data is not monotonically better
Coverage of an interval for a proportion does not improve smoothly as the sample grows. It oscillates, and there are larger samples that cover materially worse than smaller ones — a sample of twenty covers twelve points worse than a sample of nineteen.
Intervals, counted
Twenty intervals and one expected miss
The 95% belongs to the procedure, not to the interval in front of you. Twenty intervals from twenty samples make that visible in a way no definition does, and the one that misses is not a mistake.
Tests, and the second number
A p-value that is not flat is not a p-value
Under a true null, p-values are uniform. That is stronger than saying the test rejects 5% of the time, it constrains the whole distribution rather than one point of it, and it catches implementation errors that a rejection rate sails past.
Tests, and the second number
What a p-value does not say
The same p of 0.04 corresponds to a large effect in ten observations and a negligible one in two thousand. A p-value alone cannot be interpreted, and the number that makes it interpretable is almost never printed beside it.
Corrections, and what each controls
What the correction corrects
Twenty tests of true nulls produce at least one false positive 64% of the time, and the closed form and the count agree. Bonferroni holds it at 5% and Holm holds it at 5% while finding more. Nobody should still be using Bonferroni.
Reversals that are not errors
Simpson's reversal is a region, not a table
The treatment wins in both groups and loses overall. That is normally shown with one famous table, which cannot answer the two questions a reader has — how often, and how large. Swept, it turns out to occupy 31% of the allocation space.
Reversals that are not errors
What a positive test is worth
A test that is 90% sensitive and 95% specific sounds accurate. For a condition affecting one person in a thousand, 98% of its positive results are wrong, and a worse test on a commoner condition beats a better test on a rare one.
Regression, and what the summary hides
The line that one point drew
A single observation among twenty-one reverses the sign of a fitted relationship. Its leverage is known from its x value before the outcome is looked at, so this is a property of the design rather than a surprise in the data.
All 436 essays, grouped by field — or search them, or see what arrived most recently.
Threads running through
themes, not chapters
Count it, do not claim it
An interval that says 95% is making a statement about a procedure, and a procedure can be run ten thousand times. Every coverage figure on this site is a count — for a proportion it is an exact sum over the whole sample space, so the answer carries no simulation noise at all.
One run is an anecdote
A figure showing a single simulation is showing one draw from a distribution of figures. Where the claim is about behaviour rather than about one dataset, the figure runs across many seeds and reports what held for all of them — and where it shows one run, it shows twenty of them side by side.
Two routes to a number
Every simulation here has a closed form beside it and every closed form has a simulation. A distribution function and its quantile must invert each other to ten digits; an exact coverage sum and a Monte Carlo count must agree. Neither route can confirm itself, and they share no arithmetic.
The tail is where it is read
A p-value, a control limit and a risk figure are all tail statements, and the tail is exactly where every approximation in this subject is worst. The central limit theorem converges in the middle long before it converges where anyone looks.
The second number
A p-value cannot be interpreted alone. The same 0.04 corresponds to a large effect at ten observations and a negligible one at two thousand, and it means nothing at all until you know how many analyses were available to produce it.
Which rate is being controlled
Almost every procedure in this subject promises to hold some quantity at 5%, and they are not the same quantity. A familywise rate, a false discovery rate, a coverage, a per-look error rate: each is a different promise, and a method that keeps one can break another by a wide margin without doing anything wrong.
The rule is part of the result
The same observations mean different things depending on how they were going to be collected and when they were going to be looked at. A stopping rule changes a p-value without changing a number in the dataset, and a prior changes an interval's endpoints while leaving the likelihood untouched.
The symptom is absence
The defects that survive longest are the ones with nothing to look at. A balancing rule that removes exactly none of an interaction, a walk that is uniform on half a reference distribution for ever, a criterion that never sees the term it is missing: each of them passes every check that asks whether what is there is right, because the failure is that something is not.
Reversals that are nobody's mistake
Simpson's paradox, regression to the mean and the winner's curse all arise from correct arithmetic applied honestly. Each is shown as a region of a parameter space rather than one famous table, so the questions of how often and how large have answers.