Consider now the defining relation for the conditional probability P(An|B), where the Ai are mutually exclusive and their union is the entire sample space. Substitution of P(An)P(B|An)
in the numerator of equation (4) and substitution of the right-hand
side of the law of total probability in the denominator yields a result
known as Bayes’s theorem (after the 18th-century English clergyman
Thomas Bayes) or the law of inverse probability:
As an example, suppose that two balls are drawn without replacement from an urn containing r red and b black balls. Let A be the event “red on the first draw” and B the event “red on the second draw.” From the obvious relations P(A) = r/(r + b) = 1 − P(Ac), P(B|A) = (r − 1)/(r + b − 1), P(B|Ac) = r/(r + b − 1), and Bayes’s theorem, it follows that the probability of a red ball on the first draw given that the second one is known to be red equals (r − 1)/(r + b − 1). A more interesting and important use of Bayes’s theorem appears below in the discussion of subjective probabilities.
and
for any real numbers a and b. If Y is a second random variable defined on the same sample space as X and taking the values y1, y2,…, yn, the function of two variables h(xi, yj) = P{X = xi, Y = yj} is called the joint distribution of X and Y. Since {X = xi} = ∪j{X = xi, Y = yj}, and this union consists of disjoint events in the sample space,
Often f is called the marginal distribution of X to emphasize its relation to the joint distribution of X and Y. Similarly, g(yj) = ∑ih(xi, yj) is the (marginal) distribution of Y. The random variables X and Y are defined to be independent if the events {X = xi} and {Y = yj} are independent for all i and j—i.e., if h(xi, yj) = f(xi)g(yj) for all i and j. The joint distribution of an arbitrary number of random variables is defined similarly.
Suppose two dice are thrown. Let X denote the sum of the numbers appearing on the two dice, and let Y denote the number of even numbers appearing (see the table). The possible values of X are 2, 3,…, 12, while the possible values of Y are 0, 1, 2. Since there are 36 possible outcomes for the two dice, the accompanying table giving the joint distribution h(i, j) (i = 2, 3,…, 12; j = 0, 1, 2) and the marginal distributions f(i) and g(j) is easily computed by direct enumeration.
For more complex experiments, determination of a complete probability distribution usually requires a combination of theoretical analysis and empirical experimentation and is often very difficult. Consequently, it is desirable to describe a distribution insofar as possible by a small number of parameters that are comparatively easy to evaluate and interpret. The most important are the mean and the variance. These are both defined in terms of the “expected value” of a random variable.
As an example, suppose that two balls are drawn without replacement from an urn containing r red and b black balls. Let A be the event “red on the first draw” and B the event “red on the second draw.” From the obvious relations P(A) = r/(r + b) = 1 − P(Ac), P(B|A) = (r − 1)/(r + b − 1), P(B|Ac) = r/(r + b − 1), and Bayes’s theorem, it follows that the probability of a red ball on the first draw given that the second one is known to be red equals (r − 1)/(r + b − 1). A more interesting and important use of Bayes’s theorem appears below in the discussion of subjective probabilities.
Random variables, distributions, expectation, and variance
Random variables
Usually it is more convenient to associate numerical values with the outcomes of an experiment than to work directly with a nonnumerical description such as “red ball on the first draw.” For example, an outcome of the experiment of drawing n balls with replacement from an urn containing black and red balls is an n-tuple that tells us whether a red or a black ball was drawn on each of the draws. This n-tuple is conveniently represented by an n-tuple of ones and zeros, where the appearance of a one in the kth position indicates that a red ball was drawn on the kth draw. A quantity of particular interest is the number of red balls drawn, which is just the sum of the entries in this numerical description of the experimental outcome. Mathematically a rule that associates with every element of a given set a unique real number is called a “(real-valued) function.” In the history of statistics and probability, real-valued functions defined on a sample space have traditionally been called “random variables.” Thus, if a sample space S has the generic element e, the outcome of an experiment, then a random variable is a real-valued function X = X(e). Customarily one omits the argument e in the notation for a random variable. For the experiment of drawing balls from an urn containing black and red balls, R, the number of red balls drawn, is a random variable. A particularly useful random variable is 1[A], the indicator variable of the event A, which equals 1 if A occurs and 0 otherwise. A “constant” is a trivial random variable that always takes the same value regardless of the outcome of the experiment.Probability distribution
Suppose X is a random variable that can assume one of the values x1, x2,…, xm, according to the outcome of a random experiment, and consider the event {X = xi}, which is a shorthand notation for the set of all experimental outcomes e such that X(e) = xi. The probability of this event, P{X = xi}, is itself a function of xi, called the probability distribution function of X. Thus, the distribution of the random variable R defined in the preceding section is the function of i = 0, 1,…, n given in the binomial equation. Introducing the notation f(xi) = P{X = xi}, one sees from the basic properties of probabilities thatand
for any real numbers a and b. If Y is a second random variable defined on the same sample space as X and taking the values y1, y2,…, yn, the function of two variables h(xi, yj) = P{X = xi, Y = yj} is called the joint distribution of X and Y. Since {X = xi} = ∪j{X = xi, Y = yj}, and this union consists of disjoint events in the sample space,
Often f is called the marginal distribution of X to emphasize its relation to the joint distribution of X and Y. Similarly, g(yj) = ∑ih(xi, yj) is the (marginal) distribution of Y. The random variables X and Y are defined to be independent if the events {X = xi} and {Y = yj} are independent for all i and j—i.e., if h(xi, yj) = f(xi)g(yj) for all i and j. The joint distribution of an arbitrary number of random variables is defined similarly.
Suppose two dice are thrown. Let X denote the sum of the numbers appearing on the two dice, and let Y denote the number of even numbers appearing (see the table). The possible values of X are 2, 3,…, 12, while the possible values of Y are 0, 1, 2. Since there are 36 possible outcomes for the two dice, the accompanying table giving the joint distribution h(i, j) (i = 2, 3,…, 12; j = 0, 1, 2) and the marginal distributions f(i) and g(j) is easily computed by direct enumeration.
Joint distribution of
X and
Y
| i | row sum = g(j) |
||||||||||||
| 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | |||
| 0 | 1/36 | 0 | 1/18 | 0 | 1/12 | 0 | 1/18 | 0 | 1/36 | 0 | 0 | 1/4 | |
| j | 1 | 0 | 1/18 | 0 | 1/9 | 0 | 1/6 | 0 | 1/9 | 0 | 1/18 | 0 | 1/2 |
| 2 | 0 | 0 | 1/36 | 0 | 1/18 | 0 | 1/12 | 0 | 1/18 | 0 | 1/36 | 1/4 | |
| column sum = f(i) |
1/36 | 1/18 | 1/12 | 1/9 | 5/36 | 1/6 | 5/36 | 1/9 | 1/12 | 1/18 | 1/36 | ||
For more complex experiments, determination of a complete probability distribution usually requires a combination of theoretical analysis and empirical experimentation and is often very difficult. Consequently, it is desirable to describe a distribution insofar as possible by a small number of parameters that are comparatively easy to evaluate and interpret. The most important are the mean and the variance. These are both defined in terms of the “expected value” of a random variable.