Drawing Random Samples From Statistical Distributions

Drawing Random Samples From Statistical
Distributions
Paul E. Johnson 1 2
1 Department of Political Science
2 Center for Research Methods and Data Analysis, University of Kansas
2011
Descriptive
K.U.

Where do random samplings come from
Analytical solutions for some workable distributions
Approximate computational solutions for the rest
Descriptive
K.U.

Where Do We Start?
Assume we have formulae for distributions from which we
want to draw samples
Assume we have a random generator that can give us random
integers on [0, MAX]
Assume that the random generator is a good one, either
MT19937 or one of L’Ecuyer’s parallel stream generators.
Of course, do whatever book keeping necessary to assure
perfect replication
Descriptive
K.U.

PDF
Probability Density
Function, f (x)
PDF
Note small letter
used for PDF
f(x)
0.0 0.1 0.2 0.3 0.4
−3 −2 −1 0 1 2 3
x
Descriptive
K.U.

CDF: Always S-Shaped
Cumulative
Distribution
Function,
F (x)
CDF: Area on
left of point x
F (x) =
ˆ x
−∞
F(x)
f (e)de
0.0 0.2 0.4 0.6 0.8 1.0
−3 −2 −1 0 1 2 3
Descriptive
Note
CAPITAL
letter used for
CDF
CDF is always “S shaped”
x
Notation Confusing because x in F (x) is the
upper limit
K.U.

The U[0,1] is Obtained Easily from Generator
Draw from Uniform on 0, 1:
x ∼ random integer from [0, MAX ] (1)
y =
x
MAX
(2)
So, like Bogart said in the end of Casablanca, “We’ll always
have the Uniform Distribution”
Descriptive
K.U.

Other Distributions More Difficult
Inversion method:
Easiest to conceptualize
Works easily if we can calculate “quantiles” (meaning the CDF
is calculatable).
Many propose use of approximate CDF
Rejection Sampling
Find some other similar PDF that is easier to calculate
Use algorithm to test “candidates” and keep ones that fit
Composition
MCMC, and others
Descriptive
K.U.

Inversion
Consider a CDF.
What does the left
hand side mean?
Fraction of cases
smaller than
that point.
Think
“backwards” to
find x that
corresponds.
F(x)
0 1
0.21 0.58 0.82
−0.8
x
0.2
0.9
Descriptive
K.U.

Concept behind Inversion
An “equally likely”
draw from f (x)
would have this
property:
All points on the
vertical axis
between [0,1]
are going to be
equally likely.
Right?
F(x)
0 1
0.21 0.58 0.82
−0.8
x
0.2
0.9
(Otherwise, a
randomly drawn x
wouldn’t really be
random, eh?)
Descriptive
K.U.

Inversion Algorithm
Inversion method
draw a random
u ∼Uniform[0, 1]
“Think
Backwards” to
get
corresponding
x = F −1 (u)
F(x)
0 1
0.21 0.58 0.82
−0.8
0.2
0.9
Collect a lot of
those, and you’ve
got a sample from
f (x)
x
Descriptive
K.U.

Logistic: Easy Inversion
Recall the Logistic PDF, with “location” parameter µ and “scale”
parameter σ:
f (x) = 1 exp(− x−µ
σ )
σ (1 + exp(− x−µ
(3)
σ ))2
In the usual cases we discuss, µ = 0 and σ = 1.0, so the pdf in
question is simpler.
And CDF:
F (x) =
x−µ
exp(
σ )
1 + exp( x−µ
σ ) = 1
. (4)
1 + e−(x−µ)/σ Descriptive
K.U.

Calculate Logistic Inverse CDF
Figure for each probability density output value y, find the x
that corresponds to it via F:
y =
1
1 + e −(x−µ)/σ
Through the same simple algebra used in derivation of
Logistic Regression
[ ]
y
ln = (x − µ)/σ
1 − y
So
[ ]
y
x∗ = µ + σ · ln
1 − y
Descriptive
K.U.

Use That For Inversion Sampling
Draw u ∼ U[0, 1]
[ ]
Calculate x∗ = µ + σ · ln u
1−u
And, as they say on Shampoo instructions, Repeat.
Descriptive
K.U.

Limitations of Inversion
Inversion is only practical when we have a formula for F −1
that is easy to calculate.
Logistic distribution, for example, has an easy formula.
Normal, and many others, DO NOT.
So we must either
Approximate F −1 in order to conduct inversion
Find some other algorithm.
Descriptive
K.U.

Rejection Sampling*
ϕ(x): The pdf from which we want to draw a sample
ϕ(x): is some complicated formula, we can’t calculate its
CDF or inverse CDF. That means we have no obvious way to
sample from ϕ(x)
But we can calculate the PDF at any given point, and that
turns out to be the magic bullet.
*Notation Change, can’t remember why I changed from f to ϕ.
Descriptive
K.U.

General Idea behind Rejection Sampling
r(x) is different from ϕ(x) in some undertandable way.
So, draw samples from r(x)
then keep the ones you need.
How do you know when to throw away a draw from r(x)?
That’s the trick!
Descriptive
K.U.

Start with an Easy Case
Suppose
When x < 0, r(x) = ϕ(x).
When x ≥ 0, r(x) = 1.1 · ϕ(x)
For now, don’t worry if such an r(x) exists, because it
doesn’t. But it really makes the point clear.
Draw a “candidate” random number x∗ from r. Should we
keep it?
Descriptive
K.U.

If x∗ < 0, accept it
as a representation
of ϕ(x)
Because when
x < 0, r and ϕ
coincide.
probability density
0.0 0.2 0.4 0.6 0.8
Throw away this fraction
−2 −1 0 1 2 3
x
If x∗ ≥ 0.
Most of the time we want to keep that observation, since ϕ
and r coincide most of the time.
Where r and ϕ “overlap”, we want to keep x∗
That happens with probability
ϕ(x∗)/r(x∗) = ϕ(x∗)/(1.1 ∗ ϕ(x∗)) = 1/1.1
So, with probability 1/1.1 = 0.9090909 . . ., we keep x∗
Otherwise, throw it away and draw another.

More Realistic Case
r(x) is always
bigger than ϕ(x)
(by definition)
But if we can take
a draw from r(x), it
might be something
like ϕ(x) would be.
f(x)
0.0 0.2 0.4 0.6 0.8 1.0
ϕ
r
0.0 0.5 1.0 1.5 2.0 2.5 3.0
x
Descriptive
K.U.

Check Out The Size of That Gap!
The probability of
drawing x∗ = 1.9
can be calculated
from r(x) and ϕ(x)
Keep x∗ with
probability
ϕ(1.9)/r(1.9).
Amounts to
“throwing away” a
“gap sized fraction”
of candidate draws
equal to 1.9
f(x)
0.0 0.2 0.4 0.6 0.8 1.0
r(1.9)
gap= 0.09
ϕ(1.9)
0.0 0.5 1.0 1.5 2.0 2.5 3.0
x
Descriptive
K.U.

Realism and Rejection
This procedure
wastes
computational
effort
“Works” even if r is
not like ϕ at all,
but is just more
probability density
0.0 0.5 1.0 1.5
wasteful
0.0 0.2 0.4 0.6 0.8 1.0
If we draw a candidate x∗ = 0.2, we are likely to keep it
x
r(x) = 1.53
ϕ(x) = β(1.2, 1.9)
If we draw a candidate x∗ = 0.9, we are almost always going
to throw it away because r(0.9)/ϕ(0.9) is very large.
Descriptive
K.U.