Worked Example 2 — How much of your result you did not cause
The case. A litigation paralegal, panic before oral argument. He has kept a log: twelve readings of S over eight weeks, with mean μ = 66.5 and SD σ = 12.0. The week-to-week test–retest correlation in his log is r = 0.62. He arrives for the session on a bad week and his single baseline reading is X₁ = 88.
You run a clean sweep, find a driver, make the change, and he leaves visibly different. One week later he reads X₂ = 74. Fourteen points. What did you do?
Step 1 — Derive what would have happened if you had done nothing
Model the pair of readings (X₁, X₂) as bivariate normal with common mean μ, common SD σ, and correlation r. Standardise:
z₁ = (X₁ − μ)/σ , z₂ = (X₂ − μ)/σ
For the standard bivariate normal, the conditional distribution of z₂ given z₁ is
z₂ | z₁ ~ N( r·z₁ , 1 − r² )
(The full derivation is Problem 10; do it once by hand and you will never again mistake regression to the mean for an effect.) Taking expectations and un-standardising:
E[X₂ | X₁ = x] = μ + σ·r·(x − μ)/σ = μ + r(x − μ)
Substituting:
E[X₂ | X₁ = 88] = 66.5 + 0.62 × (88 − 66.5)
= 66.5 + 0.62 × 21.5
= 66.5 + 13.33
= 79.83
Expected drop with no intervention whatsoever: 88 − 79.83 = 8.17 points.
He came in on a bad week. Bad weeks are followed by ordinary weeks. Two-thirds of your fourteen points were owed before you opened your mouth.
Step 2 — Ask whether the remainder is even measurable
Attributable to intervention = 14.0 − 8.17 = 5.83
The conditional SD of a single follow-up reading:
SD(X₂ | X₁) = σ√(1 − r²) = 12 × √(1 − 0.3844) = 12 × √0.6156 = 12 × 0.7846 = 9.42
z = 5.83 / 9.42 = 0.62
A one-tailed p of roughly 0.27. Your session and a coin are, on this evidence, indistinguishable.
Step 3 — Decompose the variance, and find out what averaging can and cannot buy
Write a single reading as
X = μ + T + O + E
with T the stable charge of this memory (variance τ²), O the occasion effect (variance ω²), E the within-session instrument noise (variance ε²). Then:
τ² + ω² + ε² = σ² = 144
From Worked Example 1's method, within-session noise gives ε² = 15. The week-apart correlation is the proportion of variance shared across occasions — the stable part:
r = τ²/σ² → τ² = 0.62 × 144 = 89.28
ω² = 144 − 89.28 − 15 = 39.72
Now the crucial move. In a pre/post difference on the same person and the same memory, T cancels — it is the same T on both sides. With n baseline occasions and m follow-up occasions:
Var(Δ) = (ω² + ε²)/n + (ω² + ε²)/m = 54.72 × (1/n + 1/m)
For a single reading each side:
Var(Δ) = 54.72 × 2 = 109.44 , SE(Δ) = 10.461
Minimum detectable effect at 80% power, two-tailed α = 0.05, where the constant is z₀.₀₂₅ + z₀.₂₀ = 1.96 + 0.84 = 2.80:
MDE = 2.80 × 10.461 = 29.3
A one-reading-before, one-reading-after design can only certify effects of about thirty points or more. Four occasions each side:
Var(Δ) = 54.72 × (1/4 + 1/4) = 27.36 , SE = 5.231 , MDE = 2.80 × 5.231 = 14.6
And notice what averaging within a single session does. Var of the mean of m within-session readings is τ² + ω² + ε²/m; going from m = 1 to m = 3 takes 144 to 89.28 + 39.72 + 5 = 134.0, an SD of 11.58 against 12.0. Three and a half per cent. Taking the reading three times in the room, which feels diligent, buys you almost nothing, because the thing making your number wobble is not the instrument — it is the day. Only separate days average away days.
The consequence, which is the whole point of the chapter
The within-session sweep in Worked Example 1 had a detection threshold of 10.7. The across-week durability claim has one of 29.3. Same client, same instrument, same scale, differing by a factor of nearly three — and the entire difference is which sources of variance cancel in your design.
So: finding the driver is cheap and defensible. Claiming the change held is expensive and, as ordinarily practised, unearned. A practitioner running single-reading before-and-afters is operating an instrument that cannot see anything smaller than a thirty-point effect, which means every modest, honest, real improvement they produce is invisible to them, and every regression to the mean looks like a triumph. The field's confidence and the field's evidence are inversely related, and this arithmetic is why.
The practical response is not despair; it is to stop making small claims. Aim the work at drivers whose within-session Δ exceeds 25, insist on at least three separated baseline readings before you touch anything, and say "unverified" about everything else. The client is well served by a practitioner who knows which of their claims are certified.