Sampling Without Replacement Should Be Called Sampling

The idiom “Regression to the mean” is well-understood by the statistics spectator, but as you learn more about probability it makes less sense. Let \(X_1, \dots, X_n\) be i.i.d. and I have a hypothesis that the mean \(\mu = \mu_0\). For concreteness, suppose I am grading a final exam and I assume the average is a \(70\%\).

Under the “null” that \(\mu = \mu_0\) I observe that \(X_1, \dots, X_t\) are all less than \(\mu_0\). The adage “Regression to the mean” suggests that \(X_{t+1}\) should be larger since the mean is above the realizations. That is if I graded the first \(t\) students and they all performed poorly, then the next student should perform better than average.

However, under the i.i.d sampling model, this is false. The students are independent and thus \[ \mathsf P(X_{t+1} \mid X_1, \dots, X_t < \mu_0) = \mathsf P(X_{t+1}). \] Independence breaks “Regression to the mean”. Under the model of sampling without replacement the model remains true.

In what situation, where I’m interested in estimating the count of black marbles in an urn, would I take out a marble and then put it back in the urn? The convention that the default is sampling with replacement illustrates the tension between simple theory and practical use.