Introduction to Probability and Statistics - Universidad Carlos III de ...

Recommended reading• CM Grinstead and JM Snell (1997).Available from:Introduction to Probability, AMS.http://www.dartmouth.edu/~chance/teaching_aids/books_articles/probability_book/pdf.html• Online Statistics:online course at:An Interactive Multimedia Course of Study is a goodhttp://onlinestatbook.com/• MP Wiper (2006). Here are some notes on probability from an elementarycourse.http://halweb.uc3m.es/esp/Personal/personas/mwiper/docencia/Spanish/Doctorado_EEMC/probability_class.pdfProbability and Statistics

ProbabilityChance is a part of our everyday lives. Everyday we make judgements basedon probability:• There is a 90% chance Real Madrid will win tomorrow.• There is a 1/6 chance that a dice toss will be a 3.Probability Theory was developed from the study of games of chance by Fermatand Pascal and is the mathematical study of randomness. This theory dealswith the possible outcomes of an event and was put onto a firm mathematicalbasis by Kolmogorov.Probability and Statistics

The Kolmogorov axiomsKolmogorovFor a random experiment with sample space Ω, then a probability measureP is a function such that1. for any event A ∈ Ω, P (A) ≥ 0.2. P (Ω) = 1.3. P (∪ j∈J A j ) = ∑ j∈J P (A j) if {A j : j ∈ J} is a countable set ofincompatible events.Probability and Statistics

Laws of probabilityThe basic laws of probability can be derived directly from set theory and theKolmogorov axioms. For example, for any two events A and B, we have theaddition law,P (A ∪ B) = P (A) + P (B) − P (A ∩ B).

Laws of probabilityThe basic laws of probability can be derived directly from set theory and theKolmogorov axioms. For example, for any two events A and B, we have theaddition law,P (A ∪ B) = P (A) + P (B) − P (A ∩ B).ProofA = (A ∩ B) ∪ (A ∩ ¯B) soP (A) = P (A ∩ B) + P (A ∩ ¯B) and similarly for B. Also,A ∪ B = (A ∩ ¯B) ∪ (B ∩ Ā) ∪ (A ∩ B) soP (A ∪ B) = P (A ∩ ¯B) + P (B ∩ Ā) + P (A ∩ B)= P (A) − P (A ∩ B) + P (B) − P (A ∩ B) + P (A ∩ B)= P (A) + P (B) − P (A ∩ B).Probability and Statistics

Interpretations of probabilityThe Kolmogorov axioms provide a mathematical basis for probability but don’tprovide for a real life interpretation. Various ways of interpreting probability inreal life situations have been proposed.• The classical interpretation.• Frequentist probability.• Subjective probability.• Other approaches; logical probability and propensities.Probability and Statistics

Classical probabilityBernoulliThis derives from the ideas of Jakob Bernoulli (1713) contained in the principleof insufficient reason (or principle of indifference) developed by Laplace(1812) which can be used to provide a way of assigning epistemic or subjectiveprobabilities.Probability and Statistics

The principle of insufficient reasonIf we are ignorant of the ways an event can occur (and therefore have noreason to believe that one way will occur preferentially compared to another),the event will occur equally likely in any way.Thus the probability of an event is the coefficient between the number offavourable cases and the total number of possible cases.This is a very limited definition and cannot be easily applied in infinitedimensional or continuous sample spaces.Probability and Statistics

Subjective probabilityRamseyA different approach uses the concept of ones own probability as a subjectivemeasure of ones own uncertainty about the occurrence of an event. Thus,we may all have different probabilities for the same event because we all havedifferent experience and knowledge. This approach is more general than theother methods as we can now define probabilities for unrepeatable experiments.Subjective probability is studied in detail in Bayesian Statistics.Probability and Statistics

Other approachesKeynes• Logical probability was developed by Keynes (1921) and Carnap (1950)as an extension of the classical concept of probability. The (conditional)probability of a proposition H given evidence E is interpreted as the (unique)degree to which E logically entails H.Probability and Statistics

Popper• Under the theory of propensities developed by Popper (1957), probabilityis an innate disposition or propensity for things to happen. Long runpropensities seem to coincide with the frequentist definition of probabilityalthough it is not clear what individual propensities are, or whether theyobey the probability calculus.Probability and Statistics

Conditional probability and independenceThe probability of an event B conditional on an event A is defined asP (B|A) =P (A ∩ B).P (A)This can be interpreted as the probability of B given that A occurs.Two events A and B are called independent if P (A ∩ B) = P (A)P (B) orequivalently if P (B|A) = P (B) or P (A|B) = P (A).Probability and Statistics

The multiplication lawA restatement of the conditional probability formula is the multiplication lawP (A ∩ B) = P (B|A)P (A).Example 1What is the probability of getting two cups in two draws from a Spanish packof cards?Write C i for the event that draw i is a cup for i = 1, 2. Enumerating allthe draws with two cups is not entirely trivial. However, the conditionalprobabilities are easy to calculate:P (C 1 ∩ C 2 ) = P (C 2 |C 1 )P (C 1 ) = 939 × 1040 = 3 52 .The multiplication law can be extended to more than two events. For example,P (A ∩ B ∩ C) = P (C|A, B)P (B|A)P (A).Probability and Statistics

The birthday problemExample 2What is the probability that among n students in a classroom, at least twowill have the same birthday?http://webspace.ship.edu/deensley/mathdl/stats/Birthday.html

The birthday problemExample 2What is the probability that among n students in a classroom, at least twowill have the same birthday?http://webspace.ship.edu/deensley/mathdl/stats/Birthday.htmlThe solution is not obvious but can be solved using conditional probability. Letb i be the birthday of student i, for i = 1, . . . , n. Then it is easiest to calculatethe probability that all birthdays are distinctP (b 1 ≠ b 2 ≠ . . . ≠ b n ) = P (b n /∈ {b 1 , . . . , b n−1 }|b 1 ≠ b 2 ≠ . . . b n−1 ) ×P (b n−1 /∈ {b 1 , . . . , b n−2 }|b 1 ≠ b 2 ≠ . . . b n−2 ) × · · ·×P (b 3 /∈ {b 1 , b 2 }|b 1 ≠ b 2 )P (b 1 ≠ b 2 )Probability and Statistics

Now clearly,P (b 1 ≠ b 2 ) = 364365 , P (b 3 /∈ {b 1 , b 2 }|b 1 ≠ b 2 ) = 363365and similarlyfor i = 3, . . . , n.P (b i /∈ {b 1 , . . . , b i−1 }|b 1 ≠ b 2 ≠ . . . b i−1 ) = 366 − i365Thus, the probability that at least two students have the same birthday is, forn < 365,1 − 364365 × · · · × 366 − n 365!=365 365 n (365 − n)! .For n = 23, this probability is greater than 0.5 and for n > 50, it is virtuallyone.Probability and Statistics

The law of total probabilityThe simplest version of this rule is the following.Theorem 1For any two events A and B, thenP (B) = P (B|A)P (A) + P (B|Ā)P (Ā).We can also extend the law to the case where A 1 , . . . , A n form a partition. Inthis case, we haven∑P (B) = P (B|A i )P (A i ).i=1Probability and Statistics

Theorem 2For any two events A and B, thenBayes theoremP (A|B) =P (B|A)P (A).P (B)Supposing that A 1 , . . . , A n form a partition, using the law of total probability,we can write Bayes theorem asP (A j |B) = P (B|A j)P (A j )∑ ni=1 P (B|A i)P (A i )for j = 1, . . . , n.Probability and Statistics

The Monty Hall problemExample 3The following statement of the problem was given in a column by Marilyn vosSavant in a column in Parade magazine in 1990.Suppose you’re on a game show, and you’re given the choice of three doors:Behind one door is a car; behind the others, goats. You pick a door, say No.1, and the host, who knows what’s behind the doors, opens another door,say No. 3, which has a goat. He then says to you, “Do you want to pickdoor No. 2?” Is it to your advantage to switch your choice?Probability and Statistics

Simulating the gameHave a look at the following web page.http://www.stat.sc.edu/~west/javahtml/LetsMakeaDeal.html

Simulating the gameHave a look at the following web page.http://www.stat.sc.edu/~west/javahtml/LetsMakeaDeal.htmlUsing Bayes theoremhttp://en.wikipedia.org/wiki/Monty_Hall_problemProbability and Statistics

Random variablesA random variable generalizes the idea of probabilities for events. Formally, arandom variable, X simply assigns a numerical value, x i to each event, A i ,in the sample space, Ω. For mathematicians, we can write X in terms of amapping, X : Ω → R.Random variables may be classified according to the values they take as• discrete• continuous• mixedProbability and Statistics

Discrete variablesDiscrete variables are those which take a discrete set range of values, say{x 1 , x 2 , . . .}. For such variables, we can define the cumulative distributionfunction,F X (x) = P (X ≤ x) = ∑P (X = x i )i,x i ≤xwhere P (X = x) is the probability function or mass function.For a discrete variable, the mode is defined to be the point, ˆx, with maximumprobability, i.e. such thatP (X = x) < P (X = ˆx)for all x ≠ ˆx.Probability and Statistics

MomentsFor any discrete variable, X, we can define the mean of X to beµ X = E[X] = ∑ ix i P (X = x i ).Recalling the frequency definition of probability, we can interpret the mean asthe limiting value of the sample mean from this distribution. Thus, this is ameasure of location.In general we can define the expectation of any function, g(X) asE[g(X)] = ∑ ig(x i )P (X = x i ).In particular, the variance is defined asσ 2 = V [X] = E [ (X − µ X ) 2]and the standard deviation is simply σ = √ σ 2 . This is a measure of spread.Probability and Statistics

Chebyshev’s inequalityIt is interesting to analyze the probability of being close or far away from themean of a distribution. Chebyshev’s inequality provides loose bounds whichare valid for any distribution with finite mean and variance.Theorem 3For any random variable, X, with finite mean, µ, and variance, σ 2 , then forany k > 0,P (|X − µ| ≥ kσ) ≤ 1 k 2.Thus, we know that P (µ − √ 2σ ≤ X ≤ µ + √ 2σ) ≥ 0.5 for any variable X.Probability and Statistics

ProofP (|X − µ| ≥ kσ) = P ( (X − µ) 2 ≥ k 2 σ 2)( (X ) 2 − µ= P≥ 1)kσ[= E≤]I( ) X−µ 2≥1kσ[ (X ) ] 2 − µE= 1 kσ k 2where I is an indicator functionProbability and Statistics

Important discrete distributionsThe binomial distributionLet X be the number of heads in n independent tosses of a coin such thatP (head) = p. Then X has a binomial distribution with parameters n and pand we write X ∼ BI(n, p). The mass function isP (X = x) =(nx)p x (1 − p) n−x for x = 0, 1, 2, . . . , n.The mean and variance of X are np and np(1 − p) respectively.Probability and Statistics

The geometric distributionSuppose that Y is defined to be the number of tails observed before the firsthead occurs for the same coin. Then Y has a geometric distribution withparameter p, i.e. Y ∼ GE(p) andP (Y = y) = p(1 − p) y for y = 0, 1, 2, . . .The mean any variance of X are 1−ppand 1−pp 2respectively.Probability and Statistics

The negative binomial distributionA generalization of the geometric distribution is the negative binomialdistribution. If we define Z to be the number of tails observed beforethe r’th head is observed, then Z ∼ N B(r, p) andP (Z = z) =(r + z − 1The mean and variance of X are r 1−ppz)p r (1 − p) z for z = 0, 1, 2, . . .and r 1−pp 2respectively.The negative binomial distribution reduces to the geometric model for the caser = 1.Probability and Statistics

The hypergeometric distributionSuppose that a pack of N cards contains R red cards and that we deal n cardswithout replacement. Let X be the number of red cards dealt. Then X hasa hypergeometric distribution with parameters N, R, n, i.e. X ∼ HG(N, R, n)and( ) ( )R N − RxP (X = x) =n − x) for x = 0, 1, . . . , n.(NnExample 4In the Primitiva lottery, a contestant chooses 6 numbers from 1 to 49 and 6numbers are drawn without replacement. The contestant wins the grand prizeif all numbers match. The probability of winning is thus)P (X = x) =(66) (430(496) = 6!43!49!=113983816 .Probability and Statistics

What if N and R are large?For large N and R, then the factorials in the hypergeometric probabilityexpression are often hard to evaluate.Example 5Suppose that N = 2000 and R = 500 and n = 20 and that we wish to findP (X = 5). Then the calculation of 2000! for example is very difficult.

What if N and R are large?For large N and R, then the factorials in the hypergeometric probabilityexpression are often hard to evaluate.Example 5Suppose that N = 2000 and R = 500 and n = 20 and that we wish to findP (X = 5). Then the calculation of 2000! for example is very difficult.Theorem 4Let X ∼ HG(N, R, n) and suppose that R, N → ∞ and R/N → p. ThenP (X = x) →(nx)p x (1 − p) n−x for x = 0, 1, . . . , n.Probability and Statistics

Proof) (N − R)) (N − n)(Rx(nxP (X = x) =n − x) =R − x)=→(nx(nx( (N NnR)R!(N − R)!(N − n)!(R − x)!(N − R − n + x)!N!) R x (N − R) n−x ( )nN n → p x (1 − p) n−xxIn the example,p ( = ) 500/2000 = 0.25 and using a binomial approximation,20P (X = 5) ≈ 0.2555 0.75 15 = 0.2023. The exact answer, from Matlabis 0.2024.Probability and Statistics

The Poisson distributionAssume that rare events occur on average at a rate λ per hour. Then we canoften assume that the number of rare events X that occur in a time periodof length t has a Poisson distribution with parameter (mean and variance) λt,i.e. X ∼ P(λt). ThenP (X = x) = (λt)x e −λtx!for x = 0, 1, 2, . . .

The Poisson distributionAssume that rare events occur on average at a rate λ per hour. Then we canoften assume that the number of rare events X that occur in a time periodof length t has a Poisson distribution with parameter (mean and variance) λt,i.e. X ∼ P(λt). ThenP (X = x) = (λt)x e −λtx!for x = 0, 1, 2, . . .Formally, the conditions for a Poisson distribution are• The numbers of events occurring in non-overlapping intervals areindependent for all intervals.• The probability that a single event occurs in a sufficiently small interval oflength h is λh + o(h).• The probability of more than one event in such an interval is o(h).Probability and Statistics

Continuous variablesContinuous variables are those which can take values in a continuum. For acontinuous variable, X, we can still define the distribution function, F X (x) =P (X ≤ x) but we cannot define a probability function P (X = x). Instead, wehave the density functionf X (x) =dF (x)dx .∫Thus, the distribution function can be derived from the density as F X (x) =x−∞ f X(u) du. In a similar way, moments of continuous variables can bedefined as integrals,E[X] =∫ ∞−∞xf X (x) dxand the mode is defined to be the point of maximum density.For a continuous variable, another measure of location is the median, ˜x,defined so that F X (˜x) = 0.5.Probability and Statistics

Important continuous variablesThe uniform distributionThis is the simplest continuous distribution. A random variable, X, is said tohave a uniform distribution with parameters a and b iff X (x) = 1b − afor a < x < b.In this case, we write X ∼ U(a, b) and the mean and variance of X are a+b2and (b−a)212respectively.Probability and Statistics

The exponential distributionRemember that the Poisson distribution models the number of rare eventsoccurring at rate λ in a given time period. In this scenario, consider thedistribution of the time between any two successive events. This is anexponential random variable, Y ∼ E(λ), with density functionf Y (y) = λe −λy for y > 0.The mean and variance of X are 1 λ and 1 respectively.λ 2Probability and Statistics

The normal distributionThis is probably the most important continuous distribution. A randomvariable, X, is said to follow a normal distribution with mean and varianceparameters µ and σ 2 iff X (x) = 1σ √ 2π exp (− 12σ 2(x − µ)2 )In this case, we write X ∼ N ( µ, σ 2) .for −∞ < x < ∞.• If X is normally distributed, then a + bX is normally distributed.particular, X−µσ∼ N (0, 1).In• P (|X − µ| ≥ σ) = 0.3174, P (|X − µ| ≥ 2σ) = 0.0456, P (|X − µ| ≥ 3σ) =0.0026.• Any sum of normally distributed variables is also normally distributed.Probability and Statistics

The central limit theoremOne of the main reasons for the importance of the normal distribution is thatit can be shown to approximate many real life situations due to the centrallimit theorem.Theorem 5Given a random sample of size X 1 , . . . , X n from some distribution, thenunder certain conditions, the sample mean ¯X = 1 ∑ nn i=1 X i follows a normaldistribution.Proof See later.For an illustration of the CLT, seehttp://cnx.rice.edu/content/m11186/latest/Probability and Statistics

Mixed variablesOccasionally it is possible to encounter variables which are partially discreteand partially continuous. For example, the time spent waiting for service bya customer arriving in a queue may be zero with positive probability (as thequeue may be empty) and otherwise takes some positive value in (0, ∞).Probability and Statistics

The probability generating functionFor a discrete random variable, X, taking values in some subset of the nonnegativeintegers, then the probability generating function, G X (s) is definedasG X (s) = E [ s X] ∞∑= P (X = x)s x .x=0This function has a number of useful properties:• G(0) = P (X = 0) and more generally, P (X = x) = 1 d x G(s)x! ds x | s=0 .• G(1) = 1, E[X] = dG(1)dsand more generally, the k’th factorial moment,E[X(X − 1) · · · (X − k + 1)], isE[X!](X − k)!∣= dk G(s) ∣∣∣s=1ds kProbability and Statistics

• The variance of X isV [X] = G ′′ (1) + G ′ (1) − G ′ (1) 2 .Example 7Consider a negative binomial variable, X ∼ N B(r, p).P (X = x) =E[s X ] =(r + x − 1∞∑x=0xs x (r + x − 1x∑ ∞ (= p r r + x − 1xx=0)p r (1 − p) x for z = 0, 1, 2, . . .)p r (1 − p) x)({(1 − p)s} x p=1 − (1 − p)s) rProbability and Statistics

dEdsdE=ds∣ = r 1 − ps=1prp r (1 − p)(1 − (1 − p)s) r+1= E[X]d 2 Eds 2 = r(r + 1)pr (1 − p) 2(1 − (1 − p)s) r+2∣d 2 (E ∣∣∣s=1 1 − pds 2 = r(r + 1)p( ) 2 1 − pV [X] = r(r + 1) + r 1 − pp p= r 1 − pp 2 .) 2= E[X(X − 1)]−(r 1 − p ) 2pProbability and Statistics

The probability generating function for a sum of independent variablesSuppose that X 1 , . . . , X n are independent with generating functions G i (s) fors = 1, . . . , n. Let Y = ∑ ni=1 X i. ThenG Y (s) = E [ s Y ]= E[s ∑ ni=1 X i]==n∏E [ s X ]ii=1n∏G i (s)i=1by independenceFurthermore, if X 1 , . . . , X n are identically distributed, with common generatingfunction G X (s), thenG Y (s) = G X (s) n .Probability and Statistics

Example 8Suppose that X 1 , . . . , X n are Bernoulli trials so thatP (X i = 1) = p and P (X i = 0) = 1 − p for i = 1, . . . , nThen, the probability generating function for any X i is G X (s) = 1 − p + sp.Now consider a binomial random variable, Y = ∑ ni=1 X i. ThenG Y (s) = (1 − p + sp) nis the binomial probability generating function.Probability and Statistics

Another useful property of pgfsIf N is a discrete variable taking values on the non-negative integers and withpgf G N (s) and if X 1 , . . . , X N is a sequence of independent and identicallydistributed variables with pgf G X (s), then if Y = ∑ Ni=1 X i, we haveG Y (s) = E= E[s ∑ Ni=1 X i][E= E [ G X (s) N]= G N (G X (s))[s ∑ ]]Ni=1 X i| NThis result is useful in the study of branching processes. See the course inStochastic Processes.Probability and Statistics

The moment generating functionFor any variable, X, the moment generating function of X is defined to beM X (s) = E [ e sX] .This generates the moments of X as we haveM X (s) = E[ ∞∑i=1d i M X (s)ds i ∣∣∣∣s=0= E [ X i]](sX) ii!Probability and Statistics

Example 9Suppose that X ∼ G(α, β). Thenf X (x) = βαΓ(α) xα−1 e −βx for x > 0M X (s) =dMds==∫ ∞0∫ ∞sx βαeΓ(α) xα−1 e −βx dxβ α0 Γ(α) xα−1 e −(β−s)x dx( ) α ββ − sdM αβ α=ds (β − s) α−1∣ = αs=0β = E[X]Probability and Statistics

Example 10Suppose that X ∼ N (0, 1). ThenM X (s) ====∫ ∞−∞∫ ∞−∞∫ ∞−∞∫ ∞−∞e sx 1 √2πe −x2 2 dx(1√ exp − 1 2π 2(1√ exp − 1 2π 2(1√ exp − 1 2π 2[x 2 − 2s ]) dx[x 2 − 2s + s 2 − s 2]) dx[(x − s) 2 − s 2]) dx= e s2 2 .Probability and Statistics

The moment generating function of a sum of independent variablesSuppose we have a sequence of independent variables, X 1 , X 2 , . . . , X n withmgfs M 1 (s), . . . , M n (s). Then, if Y = ∑ ni=1 X i, it is easy to see thatM Y (s) =n∏M i (s)i=1and if the variables are identically distributed with common mgf M X (s), thenM Y (s) = M X (s) n .Probability and Statistics

Example 11Suppose that X i ∼ E(λ) for i = 1, . . . , n are independent. ThenM X (s) == λ=∫ ∞0∫ ∞0λλ − s .Therefore the mgf of Y = ∑ ni=1 X i is given bye sx λe −λx dxe −(λ−s)x dxM Y (s) =( λ) nλ − swhich we can recognize as the mgf of a gamma distribution, Y ∼ G(n, λ).Probability and Statistics

Proof of the central limit theoremFor any variable, Y , with zero mean and unit variance and such that allmoments exist, then the moment generating function isM Y (s) = E[e sY ] = 1 + s22 + o(s2 ).Now assume that X 1 , . . . , X n are a random sample from a distribution withmean µ and variance σ 2 . Then, we can define the standardized variables,Y i = X i−µσ, which have mean 0 and variance 1 for i = 1, . . . , n and thenZ n = ¯X ∑ n− µσ/ √ n = i=1 Y i√ nNow, suppose that M Y (s) is the mgf of Y i , for i = 1, . . . , n. ThenM Zn (s) = M Y(s/√ n) nProbability and Statistics

and therefore,) nM Zn (s) =(1 + s22n + o(s2 /n) → e s2 2which is the mgf of a normally distributed random variable.

and therefore,) nM Zn (s) =(1 + s22n + o(s2 /n) → e s2 2which is the mgf of a normally distributed random variable.To make this result valid for variables that do not necessarily possessall their moments, then we can use essentially the same arguments butdefining the characteristic function C X (s) = E[e isX ] instead of the momentgenerating function.Probability and Statistics

Multivariate distributionsIt is straightforward to extend the concept of a random variable to themultivariate case. Full details are included in the course on MultivariateAnalysis.For two discrete variables, X and Y , we can define the joint probability functionat (X = x, Y = y) to be P (X = x, Y = y) and in the continuous case, wesimilarly define a joint density function f X,Y (x, y) such that∑ ∑P (X = x, Y = y) = 1and similarly for the continuous case.xy∑P (X = x, Y = y) = P (X = x)y∑P (X = x, Y = y) = P (Y = y)xProbability and Statistics

Conditional distributionsThe conditional distribution of Y given X = x is defined to bef Y |x (y|x) = f X,Y (x, y).f X (x)Two variables are said to be independent if for all x, y, then f X,Y (x, y) =f X (x)f Y (y) or equivalently if f Y |x (y|x) = f Y (y) or f X|y (x|y) = f X (x).We can also define the conditional expectation of Y |x to be E[Y |x] =∫yfY |x (y|x) dx.Probability and Statistics

Covariance and correlationIt is useful to obtain a measure of the degree of relation between the twovariables. Such a measure is the correlation.We can define the expectation of any function, g(X, Y ), in a similar way tothe univariate case,∫ ∫E[g(X, Y )] = g(x, y)f X,Y (x, y) dx dy.In particular, the covariance is defined asσ X,Y = Cov[X, Y ] = E[XY ] − E[X]E[Y ].Obviously, the units of the covariance are the product of the units of X andY . A scale free measure is the correlation,ρ X,Y = Corr[X, Y ] = σ X,Yσ X σ YProbability and Statistics

Properties of the correlation are as follows:• −1 ≤ ρ X,Y ≤ 1• ρ X,Y = 0 if X and Y are independent. (This is not necessarily true inreverse!)• ρ XY = 1 if there is an exact, positive relation between X and Y so thatY = a + bX where b > 0.• ρ XY = −1 if there is an exact, negative relation between X and Y so thatY = a + bX where b < 0.Probability and Statistics

Conditional expectations and variancesTheorem 6For two variables, X and Y , thenE[Y ] = E[E[Y |X]]V [Y ] = E[V [Y |X]] + V [E[Y |X]]ProofE[E[Y |X]] = E∫=∫=∫=[∫∫y∫y]yf Y |X (y|X) dy=∫f Y |X (y|x)f X (x) dx dyf X,Y (x, y) dx dyyf Y (y) dy = E[Y ]∫f X (x)yf Y |X (y|X) dy dxProbability and Statistics

Example 12A random variable X has a beta distribution, X ∼ B(α, β), iff X (x) =Γ(α + β)Γ(α)Γ(β) xα−1 (1 − x) β−1 for 0 < x < 1.The mean of X is E[X] =αα+β .Suppose now that we toss a coin with probability P (heads) = X a total of ntimes and that we require the distribution of the number of heads, Y .This is the beta-binomial distribution which is quite complicated:Probability and Statistics

P (Y = y) ====∫ 10∫ 1P (Y = y|X = x)f X (x) dx( )n0y( )n Γ(α + β)y Γ(α)Γ(β)(nyx y n−y Γ(α + β)(1 − x)∫ 10Γ(α)Γ(β) xα−1 (1 − x) β−1 dxx α+y−1 (1 − x) β+n−y−1 dx) Γ(α + β) Γ(α + y)Γ(β + n − y)Γ(α)Γ(β) Γ(α + β + n)for y = 0, 1, . . . , n.Probability and Statistics

We could try to calculate the mean of Y directly using the above probabilityfunction. However, this would be very complicated. There is a much easierway.

We could try to calculate the mean of Y directly using the above probabilityfunction. However, this would be very complicated. There is a much easierway.E[Y ] = E[E[Y |X]]= E[nX] because Y |X ∼ BI(n, X)= n αα + β .Probability and Statistics

StatisticsStatistics is the science of data analysis. This is concerned with• how to generate suitable samples of data• how to summarize samples of data to illustrate their important features• how to make inference about populations given sample data.Probability and Statistics

SamplingIn statistical problems we usually wish to study the characteristics of somepopulation. However, it is usually impossible to measure the values of thevariables of interest for all members of the population. This implies the use ofa sample.There are many possible ways of selecting a sample. Non random approachesinclude:• Convenience sampling• Volunteer sampling• Quota samplingSuch approaches can suffer from induced biases.Probability and Statistics

Random samplingA better approach is random sampling. For a population of elements, saye 1 , . . . , e N , then a simple random sample of size n selects every possible n-tupleof elements with equal probability. Unrepresentative samples can be selectedby this approach, but is no a priori bias which means that this is likely.When the population is large or heterogeneous, other random samplingapproaches may be preferred. For example:• Systematic sampling,• Stratified sampling• Cluster sampling• Multi stage samplingSampling theory is studied in more detail in Quantitative Methods.Probability and Statistics

Descriptive statisticsGiven a data sample, it is important to develop methods to summarizethe important features of the data both visually and numerically. Differentapproaches should be used for different types of data.• Categorical data:– Nominal data,– Ordinal data.• Numerical data:– Discrete data,– Continuous data.Probability and Statistics

Categorical dataCategorical data are those that take values in different categories, e.g. bloodtypes, favourite colours, etc. These data may be nominal, when the differentcategories have no inherent sense of order or ordinal, when the categories arenaturally ordered.Example 13The following table gives the frequencies of the different first movesin a chess game found on 20/02/1996 using the search engine ofhttp://www.chessgames.com/.Opening Frequency Relative frequencye4 178130 0.4794d4 125919 0.3389Nf3 32206 0.0867c4 28796 0.0776Others 6480 0.0174T otal 371531 1.0000Probability and Statistics

This is an example of a sample of nominal data. The frequency table has beenaugmented with the relative frequencies or proportions in each class. We cansee immediately that the most popular opening or modal class is e4, played innearly half the games.A nice way of visualizing the data is via a pie chart. This could be augmentedwith the frequencies or relative frequencies in each class.Othersc4Nf3e4d4Probability and Statistics

An alternative display is a bar chart which can be constructed using frequenciesor relative frequencies.18 x 104 Opening161412Frequency1086420e4 d4 Nf3 c4 OthersWhen the data are categorical, it is usual to order them from highest to lowestfrequency. With ordinal data, it is more sensible to use the ordering of theclasses.Probability and Statistics

A final approach which is good to look at but not so easy to interpret is thepictogram. The area of each image is proportional to the frequency.e4 d4 Nf3 c4 OthersProbability and Statistics

Measuring the relation between two categorical variablesOften we may record the values of two (or more) categorical variables. In suchcases we are interested in whether or not there is any relation between thevariables. To do this, we can construct a contingency table.Example 14The following data given in Morrell (1999) come from a South African studyof single birth children. At birth in 1990 it was recorded whether or not themothers received medical aid and later, in 1995 the researchers attempted totrace the children. Those children found were included in the five year groupfor further study.Children not traced Five-Year GroupHad Medical Aid 195 46No Medical Aid 979 3701590CH Morrell (1999). Simpson’s Paradox: An Example From a Longitudinal Study in South Africa. Journal of Statistics Education, 7.Probability and Statistics

Analysis of a contingency tableIn order to analyze the contingency table it is useful to first calculate themarginal totals.Children not traced Five-Year GroupHad Medical Aid 195 46 241No Medical Aid 979 370 13491174 416 1590and then to convert the original data into percentages.Children not traced Five-Year GroupHad Medical Aid .123 .029 .152No Medical Aid .615 .133 .848.738 .262 1Probability and Statistics

Then it is also possible to calculate conditional frequencies. For example, theproportion of children not traced who received medical aid is195/1174 = .123/.738 = .166.Finally, we may often wish to assess whether there exists any relation betweenthe two variables. In order to do this we can assess how many data we wouldexpect to see in each cell, assuming the marginal totals if the data really wereindependent.Children not traced Five-Year GroupHad Medical Aid 177.95 63.05 241No Medical Aid 996.05 352.95 13491174 416 1590Comparing these expected totals with the original frequencies, we could set upa formal statistical (χ 2 ) test for independence.Probability and Statistics

Simpson’s paradoxSometimes we can observe apparently paradoxical results when a populationwhich contains heterogeneous groups is subdivided. The following example ofthe so called Simpson’s paradox comes from the same study.http://www.amstat.org/publications/jse/secure/v7n3/datasets.morrell.cfmProbability and Statistics

Numerical dataWhen data are naturally numerical, we can use both graphical and numericalapproaches to summarize their important characteristics. For discrete data, wecan frequency tables and bar charts in a similar way to the categorical case.Example 15The table reports the number of previous convictions for 283 adult malesarrested for felonies in the USA taken from Holland et al (1981).# Previous convictions Frequency Rel. freq. Cum. freq. Cum. rel. freq.0 0 0.0000 0 0.00001 16 0.0565 16 0.05652 27 0.0954 43 0.15193 37 0.1307 80 0.28274 46 0.1625 126 0.44525 36 0.1272 162 0.57246 40 0.1413 202 0.71387 31 0.1095 233 0.82338 27 0.0954 260 0.91879 13 0.0459 273 0.964710 8 0.0283 281 0.992911 2 0.0071 283 1.0000> 11 0 0.0000 283 1.0000TR Holland, M Levi & GE Beckett (1981). Associations Between Violent And Nonviolent Criminality: A Canonical Contingency-TableAnalysis. Multivariate Behavioral Research, 16, 237–241.Probability and Statistics

Note that we have augmented the table with relative frequencies and cumulativefrequencies. We can construct a bar chart of frequencies as earlier but wecould also use cumulative or relative frequencies.5045403530Frequency25201510500 1 2 3 4 5 6 7 8 9 10 11 12Number of previous convictionsNote that the bars are much thinner in this case. Also, we can see that thedistribution is slightly positively skewed or skewed to the right.Probability and Statistics

Continuous data and histogramsWith continuous data, we should use histograms instead of bar charts. Themain difficulty is in choosing the number of classes. We can see the effects ofchoosing different bar widths in the following web page.http://www.shodor.org/interactivate/activities/histogram/An empirical rule is to choose around √ n classes where n is the number ofdata. Similar rules are used by the main statistical packages.It is also possible to illustrate the differences between two groups of individualsusing histograms. Here, we should use the same classes for both groups.Probability and Statistics

Example 16The table summarizes the hourly wage levels of 30 Spanish men and 25 Spanishwomen, (with at least secondary education) who work > 15 hours per week.MWInterval n i f i n i f i[300, 600) 1 .033 0 0[600, 900) 1 .033 1 .04[900, 1200) 2 .067 7 .28[1200, 1500) 5 .167 10 .4[1500, 1800) 10 .333 6 .24[1800, 2100) 8 .267 1 .04[2100, 2400) 3 .100 0 0> 2400 0 0 0 030 1 25 1J Dolado and V LLorens (2004). Gender Wage Gaps by Education in Spain: Glass Floors vs. Glass Ceilings, CEPR DP., 4203.http://www.eco.uc3m.es/temp/dollorens2.pdfProbability and Statistics

menwage✻24002100180015001200900600300✻womenf✛.4 .3 .2 .1 0 0 .1 .2 .3 .4 .5✲fThe male average wage is a little higher and the distribution of the male wagesis more disperse and asymmetric.Probability and Statistics

Histograms with intervals of different widthsIn this case, the histogram is constructed so that the area of each bar isproportional to the number of data.Example 17The following data are the results of a questionnaire to marijuana usersconcerning the weekly consumption of marijuana.g / week Frequency[0, 3) 94[3, 11) 269[11, 18) 70[18, 25) 48[25, 32) 31[32, 39) 10[39, 46) 5[46, 74) 2> 74 0Landrigan et al (1983). Paraquat and marijuana: epidemiologic risk assessment. Amer. J. Public Health, 73, 784-788Probability and Statistics

We augment the table with relative frequencies and bar widths and heights.g / week width n i f i height[0, 3) 3 94 .178 .0592[3, 11) 8 269 .509 .0636[11, 18) 7 70 .132 .0189[18, 25) 7 48 .091 .0130[25, 32) 7 31 .059 .0084[32, 39) 7 10 .019 .0027[39, 46) 7 5 .009 .0014[46, 74) 28 2 .004 .0001> 74 0 0 0 0Total 529 1We use the formulaheight = frequency/interval widthProbability and Statistics

freq/width.07✻.06.05.04.03.02.01.000 10 20 30 40 50 60 70 80✲g/weekThe distribution is very skewed to the right.Probability and Statistics

Other graphical methods• the frequency polygon. A histogram is constructed and the bars are linesare used to join each bar at the centre. Usually the histogram is thenremoved. This simulates the probability density function.• the cumulative frequency polygon. As above but using a cumulativefrequency histogram and joining at the end of each bar.• the stem and leaf plot. This is like a histogram but retaining the originalnumbers.Probability and Statistics

Sample momentsFor a sample, x 1 , . . . , x n of numerical data, then the sample mean is definedas ¯x = 1 n∑ ni=1 x i and the sample variance is s 2 = 1n−1∑ ni=1 (x i − ¯x) 2 . Thesample standard deviation is s = √ s 2 .The sample mean may be interpreted as an estimator of the population mean.It is easiest to see this if we consider grouped data say x 1 , . . . , x k where x j isobserved n j times in total and ∑ ki=1 n i = n. Then, the sample mean is¯x = 1 nk∑n j x j =j=1k∑f j x jj=1where f j is the proportion of times that x j was observed.When n → ∞, then (using the frequency definition of probability), we knowthat f j → P (X = x j ) and so ¯x → µ X , the true population mean.Sometimes the sample variance is defined with a denominator of n instead ofn − 1. However, in this case, it is a biased estimator of σ 2 .Probability and Statistics

Problems with outliers, the median and interquartile rangeThe mean and standard deviation are good estimators of location and spreadof the data if there are no outliers or if the sample is reasonably symmetric.Otherwise, it is better to use the sample median and interquartile range.Assume that the sample data are ordered so that x 1 ≤ x 2 ≤ . . . ≤ x n . Thenthe sample median is defined to be˜x =⎧⎨⎩xn+12if n is oddxn2+x n+222if n is even.For example, if we have a sample 1, 2, 6, 7, 8, 9, 11, the median is 7 and forthe sample 1, 2, 6, 7, 8, 9, 11, 12 then the median is 7.5. We can think of themedian as dividing the sample in two.Probability and Statistics

We can also define the quartiles in a similar way. The lower quartile isQ 1 = xn+1 and the upper quartile may be defined as Q 3 = x 3(n+1) where if the44fraction is not a whole number, the value should be derived by interpolation.Thus, for the sample 1, 2, 6, 7, 8, 9, 11, then Q 1 = 2 and Q 3 = 9. For thesample 1, 2, 6, 7, 8, 9, 11, 12, we have n+14= 2.25 so thatand 3(n+1)4= 6.75 soQ 1 = 2 + 0.25(6 − 2) = 3Q 3 = 9 + 0.75(11 − 9) = 10.5.A nice visual summary of a data sample using the median, quartiles and rangeof the data is the so called box and whisker plot or boxplot.http://en.wikipedia.org/wiki/Box_plotProbability and Statistics

Correlation and regressionOften we are interested in modeling the extent of a linear relationship betweentwo data samples.Example 18In a study on the treatment of diabetes, the researchers measured patientsweight losses, y, and their initial weights on diagnosis, x to see if weight losswas influenced by initial weight.X 225 235 173 223 200 199 129 242Y 15 44 31 39 6 16 21 44X 140 156 146 195 155 185 150 149Y 5 12 −3 19 10 24 −3 10In order to assess the relationship, it is useful to plot these data as a scatterplot.Probability and Statistics

We can see that there is a positive relationship between initial weight andweight loss.Probability and Statistics

The sample covarianceIn such cases, we can measure the extent of the relationship using the samplecorrelation. Given a sample, (x 1 , y 1 ), . . . , (x n , y n ), then the sample covarianceis defined to bes xy = 1 n∑(x i − ¯x)(y i − ȳ).n − 1In our example, we havei=1¯x = 1 (225 + 235 + . . . + 149)16= 181.375ȳ = 1 (15 + 44 + . . . + 10)16= 18.125s xy = 1 {(225 − 181.375)(15 − 18.125)+16(235 − 181.375)(44 − 18.125) + . . . +(149 − 181.375)(10 − 18.125)} ≈ 361.64Probability and Statistics

The sample correlationThe sample correlation isr xy = s xys x s ywhere s x and s y are the two standard deviations. This has properties similarto the population correlation.• −1 ≤ r xy ≤ 1.• r xy = 1 if y = a + bx and r xy = −1 if y = a − bx for some b > 0.• If there is no relationship between the two variables, then the correlation is(approximately) zero.In our example, we find that s 2 x ≈ 1261.98 and s 2 y ≈ 211.23 so that s x ≈ 35.52and s y ≈ 14.53. which implies that r xy = 361.6435.52×14.53≈ 0.70 indicating astrong, positive relationship between the two variables.Probability and Statistics

Correlation only measures linear relationships!High or low correlations can often be misleading.In both cases, the variables have strong, non-linear relationships. Thus,whenever we are using correlation or building regression models, it is alwaysimportant to plot the data first.Probability and Statistics

Spurious correlationCorrelation is often associated with causation. If X and Y are highly correlated,it is often assumed that X causes Y or Y causes X.

Spurious correlationCorrelation is often associated with causation. If X and Y are highly correlated,it is often assumed that X causes Y or Y causes X.Probability and Statistics

Example 19Springfield had just spent millions of dollars creating a highly sophisticated”Bear Patrol” in response to the sighting of a single bear the week before.Homer: Not a bear in sight. The ”Bear Patrol” is working like a a charmLisa: That’s specious reasoning, Dad.Homer:[uncomprehendingly] Thanks, honey.Lisa: By your logic, I could claim that this rock keeps tigers away.Homer: Hmm. How does it work? Lisa:It doesn’t work. (pause) It’s just a stupid rock!Homer: Uh-huh.Lisa: But I don’t see any tigers around, do you?Homer: (pause) Lisa, I want to buy your rock.Much Apu about nothing. The Simpsons series 7.Probability and Statistics

Example 201988 US census data showed that numbers of churches in a city was highlycorrelated with the number of violent crimes. Does this imply that havingmore churches means that there will be more crimes or that having more crimemeans that more churches are built?

Example 201988 US census data showed that numbers of churches in a city was highlycorrelated with the number of violent crimes. Does this imply that havingmore churches means that there will be more crimes or that having more crimemeans that more churches are built?Both variables are highly correlated to population.them is spurious.The correlation betweenProbability and Statistics

RegressionAn model representing an approximately linear relation between x and y iswhere ɛ is a prediction error.y = α + βx + ɛIn this formulation, y is the dependent variable whose value is modeled asdepending on the value of x, the independent variable .How should we fit such a model to the data sample?Probability and Statistics

Least squaresGaussWe wish to find the line which best fits the sample data (x 1 , y 1 ), . . . , (x n , y n ).In order to do this, we should choose the line, y = a + bx, which in some wayminimizes the prediction errors or residuals,e i = y i − (a + bx i ) for i = 1, . . . , n.Probability and Statistics

A minimum criterion would be that ∑ ni=1 e i = 0. However, many lines satisfythis, for example y = ȳ. Thus, we need a stronger constraint.The standard way of doing this is to choose to minimize the sum of squarederrors, E(a, b) = ∑ ni=1 e2 i .Theorem 7For a sample (x 1 , y 1 ), . . . , (x n , y n ), the line of form y = a+bx which minimizesthe sum of squared errors, E[a, b] = ∑ ni=1 (y i − a − bx i ) 2 is such thatb = s xys 2 xa = ȳ − b¯xProbability and Statistics

Proof Suppose that we fit the line y = a + bx. We want to minimize thevalue of E(a, b). We can recall that at the minimum,∂E∂a = ∂E∂b = 0.Now, E = ∑ ni=1 (y i − a − bx i ) 2 and therefore∂E∂a= −20 = −2n∑(y i − a − bx i ) and at the minimumi=1n∑(y i − a − bx i )i=1= −2 (nȳ − na − nb¯x)a = ȳ − b¯xProbability and Statistics

∂E∂b= −2n∑x i (y i − a − bx i )i=1(∑ n0 = −2 x i y i −n∑x i y i =i=1=b =i=1n∑x i (a + bx i )i=1)n∑x i (a + bx i )i=1n∑x i (ȳ − b¯x + bx i )i=1( n)∑= n¯xȳ + b x 2 i − n¯x 2i=1∑ ni=1 x iy i − n¯xȳ∑ ni=1 x2 i − n¯x2and at the minimum,substituting for aProbability and Statistics= ns xyns 2 x= s xys 2 x

We will fit the regression line to the data of our example on the weights ofdiabetics. We have seen earlier that ¯x = 181.375, ȳ = 18.125 , s xy = 361.64,s 2 x = 1261.98 and s 2 y = 211.23.Thus, if we wish to predict the values of y (reduction in weight) in terms of x(original weight), the least squares regression line iswherey = a + bxb = 361.641261.98 ≈ 0.287a = 18.125 − 0.287 × 181.375 ≈ −33.85The following diagram shows the fitted regression line.Probability and Statistics

Probability and Statistics

We can use this line to predict the weight loss of a diabetic given their initialweight. Thus, for a diabetic who weighed 220 pounds on diagnosis, we wouldpredict that their weight loss would be aroundŷ = −33.85 + 0.287 × 220 = 29.29 lbs.Note that we should be careful when making predictions outside the range ofthe data. For example the linear predicted weight gain for a 100 lb patientwould be around 5 lbs but it is not clear that the linear relationship still holdsat such low values.Probability and Statistics

Residual analysisLos residuals or prediction errors are the differences e i = y i − (a + bx i ). It isuseful to see whether the average prediction error is small or large. Thus, wecan define the residual variances 2 e = 1n − 1n∑i=1e 2 iand the residual standard deviation, s e = √ s 2 e.In our example, we have e 1 = 15−(−33.85+0.287×225), e 2 = 44−(−33.85+0.287 × 235) etc. and after some calculation, the residual sum of squares canbe shown to be s 2 e ≈ 123. Calculating the results this way is very slow. Thereis a faster method.Probability and Statistics

Theorem 8ē = 0s 2 r = s 2 y(1 − r2xy)Proofē = 1 n= 1 n= 1 n= 0n∑(y i − (a + bx i ))i=1n∑(y i − (ȳ − b¯x + bx i )) by definition of ai=1(∑ n(y i − ȳ) − bi=1i=1)n∑(x i − ¯x)Probability and Statistics

s 2 e ====1n − 1n∑(y i − (a + bx i )) 2i=11n − 1 (y i − (ȳ − b¯x + bx i )) 2 by definition of a1n − 1 ((y i − ȳ) − b(x i − ¯x))) 2(1 ∑ n n∑n)∑(y i − ȳ) 2 − 2b (y i − ȳ)(x i − ¯x) + b 2 (x i − ¯x) 2n − 1i=1i=1i=1= s 2 y − 2bs xy + b 2 s 2 x = s 2 y − 2 s xys xy += s 2 y − s2 xy= s 2 y(s 2 x1 −(= s 2 y 1 − s2 xys 2 xs 2 y( ) ) 2 sxy= s 2 ys x s y)s 2 x(1 − r2xy)(sxys 2 x) 2s 2 x by definition of bProbability and Statistics

This result shows thatInterpretations 2 rs 2 y= (1 − r 2 xy).Consider the problem of estimating y. If we only observe y, . . . , y n , then ourbest estimate is ȳ and the variance of our data is s 2 y.Given the x data, then our best estimate is the regression line and the residualvariance is s 2 r.Thus, the percentage reduction in variance due to fitting the regression line isR 2 = (1 − r 2 xy) × 100%In our example, r xy ≈ 0.7 so R 2 = (1 − 0.49) × 100% = 51%.Probability and Statistics

Graphing the residualsAs we have seen earlier, the correlation between two variables can be highwhen there is a strong non-linear relation.Whenever we fit a linear regression model, it is important to use residual plotsin order to check the adequacy of the fit.The regression line for the following five groups of data, from Basset et al(1986) is the same, that isy = 18.43 + 0.28xBassett, E. et al (1986). Statistics: Problems and Solutions. London: Edward ArnoldProbability and Statistics

Probability and Statistics

• The first case is a standard regression.• In the second case, we have a non-linear fit.• In the third case, we can see the influence of an outlier.• The fourth case is a regression but . . .• In the final case, we see that one point is very influential.Now we can observe the residuals.Probability and Statistics

In case 4 we see the residuals increasing as y increases.Probability and Statistics

Two regression linesSo far, we have used the least squares technique to fit the line y = a + bxwhere a = ȳ − b¯x and b = s xy.s 2 xWe could also rewrite the linear equation in terms of x and try to fit x = c+dy.Then, via least squares, we have that c = ¯x − dȳ and d = s xy.s 2 yWe might expect that these would be the same lines, but rewritingy = a + bx ⇒ x = − a b + 1 b y ≠ c + dyIt is important to notice that the least squares technique minimizes theprediction errors in one direction only.Probability and Statistics

The following example shows data on the extension of a cable, y relative toforce applied, x and the fit of both regression lines.Probability and Statistics

Regression and normalitySuppose that we have a statistical modelY = α + βx + ɛwhere ɛ ∼ N (0, σ 2 ). Then, if data come from this model, it can be shown thatthe least squares fit method coincides with the maximum likelihood approachto estimating the parameters.You will study this in more detail in the course on Regression Models.Probability and Statistics