Bennett Haselton, 2006/11/06
Definition: Given an event A, the expression p(A) refers to the probability of event A occuring.
In the first section we consider questions such as the following: If p(A) = 1/2 and p(B) = 1/3, then what is the maximum possible probability of event A or event B occurring?
| Note that in questions such as these, events A and B are not stated to be independent of each other. If events A and B are independent, then none of the reasoning in this section would apply, so keep that in mind and be sure to read probability questions carefully for mentions of the word "independent"! |
Given events A and B, you can represent the probability of the events using a diagram
similar to a Venn diagram from set theory:
Imagine that the area inside the rectangle is 1. The total area inside the circle A represents the probability of event A occurring, and the area inside circle B represents the probability of event B occurring. Thus if you were to pick a random point inside the rectangle, the probability that you would land inside circle A, is the same as the probability of event A.
Similarly, the probability that event A and event B will occur, is represented by the area
where the circles intersect:
This means that the probability that event A and event B will occur, is the same as the
probability that if you pick a random point inside the rectangle, you will land inside circle A
and circle B. Thus there is a nice correspondence between "events A and B both occurring"
and "being inside both circle A and circle B".
And the probability that event A or event B will occur, is represented by the area of the union
of the two circles:
By now you should be able to guess what comes next: The probability that event A or event B will
occur, is the same as the probability that if you pick a random point inside the rectangle, it will
land inside circle A or inside circle B.
So, to answer the original question: Suppose p(A) = 1/2 and p(B) = 1/3. What is the maximum possiblity probability of event A or event B occurring?
Stated in terms of the circles representing A and B, the question can be rephrased as follows: given a circle A with area 1/2 and a circle B with area 1/3, how can they be arranged to give the maximum value for the area of A-union-B?
Visually, now the answer is obvious: circles A and B cover the maximum combined area, when they do not overlap:
So the maximum possible area of A-union-B is 1/2 + 1/3 = 5/6, and this is the maximum possible probability of
event A or event B occurring.
| Note: these "maximum possible probability" and "minimum possible probability" questions are an unusual type of probability problem, because they allow you to change the probabilities of different events -- i.e. moving circle A and circle B apart so that their intersection, i.e. the probability of A-and-B, is zero, and their union, the probability of A-or-B, is maximized at 5/6. In the vast majority of probability questions -- e.g. "If you take two marbles from a bag, what is the probability that at least one will be red" -- the probabilities of different events is fixed, and all you can do is calculate them, not actually change them. So it's important not to get the two types of questions mixed up! |
In this case the answer was simply p(A) + p(B). But what if p(A) had been 3/4 and p(B) had been 1/2? If you had two shapes of area 3/4 and 1/2 inside a rectangle of area 1, it wouldn't be possible to arrange the shapes so that they were non-overlapping, because then they would take up a total area of 3/4 + 1/2 = 5/4, but the total area inside the rectangle is only 1. So the maximum possible area that the shapes could cover, would be to fill up the entire inside of the rectangle -- in other words, the area of their union would be 1, so the maximum probability of event A or event B occurring would be 1. To summarize:
Statement 1: Given events A and B, the maximum possible probability of event A or B occurring is:
What about the minimum possible probability of A or B?
Once again, it is much easier to solve the problem visually in terms of circles. The probability of A or B
is represented by the union of the circles A and B. How can you arrange the circles so that together they
take up the least possible amount of area? By putting one inside the other:
So here, if p(A) = 1/2 and p(B) = 1/3, the minimum possible probability of A or B is 1/2, the greater of the two. In general:
Statement 2: Given events A and B, the minimum possible probability of event A or B occurring is max{p(A), p(B)}.
Now suppose you want to find the maximum possible probability of A and B. Remember, the probability
of events A and B occurring, is represented in the probability Venn diagram by the intersection of the
circles A and B. How do you maximize the area of the intersection? Again, the answer is to put one
circle completely inside the other:
in which case the area of the intersection is the area of the smaller of the two circles. If p(A) = 1/2 and
p(B) = 1/3, then the maximum possible area of the intersection, and hence the maximum possible area
of the overlap, is p(1/3), the smaller of the two. In general:
Statement 3: Given events A and B, the maximum possible probability of events A and B both occurring is min{p(A), p(B)}.
Finally, we come to the question: What is the minimum possible probability of A and B?
Viewing the problem in terms of shapes inside the rectangle, we want to arrange shapes A and B such that
their intersection is minimized. Sometimes it is possible to arrange the shapes A and B such that they
don't overlap at all, as shown earlier:
in which case the probability of A-and-B is zero.
But this is only possible when p(A) + p(B) < 1. Otherwise there is not enough "room" inside the
rectangle to separate them, because when they are non-overlapping, they occupy a total area of p(A) + p(B),
and if p(A) + p(B) > 1 then they can't fit inside the rectangle without some overlap.
So, if A and B can't be separated completely, what is the minimum possible overlap?
Suppose you draw shapes A and B with no overlap, but with one of them jutting outside of the rectangle of
area 1. In this diagram, A is represented by one set of diagonal lines and B is represted by the other:
The portion of B that has to be moved inside the rectangle, is the portion that is currently jutting outside
of it. Since the total area of regions A and B is p(A) + p(B), that means the area jutting outside the
rectangle is p(A) + p(B) - 1 (because you subtract the area of the rectangle, which is 1).
And if you "slide" B to the left to fit it inside:
then the area that got moved inside the rectangle, is the same as the area that is now overlapping with
region A -- so the area of the overlap is p(A) + p(B) - 1. This is the minimum possible overlap between
A and B and the answer to our question.
Statement 4: Given events A and B, the minimum possible probability of A and B both occurring is:
We can summarize the results so far in the following table. The point is not to memorize this table, though, but to practice visualizing the combined probabilities in terms of overlapping circles, at which point it becomes much easier to see why the formulas work.
| Quantity to find | Formula | Visual interpretation | Diagram |
| Maximum possible value of A-or-B |
|
Two shapes have maximum area of their union when they are non-overlapping (or, if there is not enough room for them to be non-overlapping, then when they overlap as little as possible) |
|
| Minimum possible value of A-or-B | max{p(A), p(B)} | Two shapes have minimum area of their union when one is completely inside the other |
|
| Maximum possible value of A-and-B | min{p(A), p(B)} | Two shapes have maximum area of their intersection when one is completely inside the other |
|
| Minimum possible value of A-and-B |
|
Two shapes have minimum area of their intersection when they are non-overlapping (or, if there is not enough room for them to be non-overlapping, then when they overlap as little as possible) |
|
You probably noticed that:
Statement 5:
Also, note that using these formulas, given any combination of p(A), p(B), p(A and B), p(A or B), if you know three of those values, you can find the fourth one.
Practice problem: The probability of A is 1/3. The probability of A and B is 0. What is the maximum possible probability of B? (Hint: think again in terms of shapes inside a rectangle of area 1. If A takes up area 1/3 and B doesn't overlap with A, how much space could B take up?)
Practice problem: The probablity of A is 1/4. The probability of A and B is 1/4. What is the maximum possible probability of B? (Similar to the last question, except that B can now take up all the available remaining space plus a region overlapping A with area 1/4.)
Practice problem: If p(B) = 0, then what is the probability of A-or-B? (Hint: consider the area of the union of A and B, if the area of B is 0.)
Practice problem: If p(A) = 1/3, p(B) = 1/3, and p(A and B) = 1/4, then show that p(A or B) = 5/12.
Consider again the case where the probability regions for A and B do not overlap:
This means that given that A occurs, there is zero probability of B occurring, and vice versa.
The events are disjoint, or mutually exclusive. An example would be if event A corresponds to the
probability of a random person being less than 5 feet tall, and B corresponds to the probability of
that person being more than 6 feet tall -- they cannot occur at the same time. When two mutually
exclusive events are represented with shapes in a Venn diagram, the area of their union is just equal
to the sum of their individual areas. In other words:
Statement 6: If A and B are mutually exclusive events, then the probability of event A or event B occurring is p(A) + p(B).
Or consider the case where the region for B lies entirely inside A:
This means that given that B occurs, we know with 100% certainly that A occurs as well (although
not vice versa in this case). An example would be if event A corresponds to the probability of a person
being over 5 feet tall, and B being the probability of that person being over 6 feet tall -- if B is true,
then A must be true.
These are the two simplest cases of conditional probabilities -- given that you know one event
occurs, what does that tell you about the probability of another event occurring? Consider a less simple
example, in which the probabilities of events A and B are given in the following diagram:
Each of the four regions is labelled with its area, and the areas of the four regions add up to 1. So the area of each region, is also the probability of landing in that region if you were to pick a random point inside the rectangle.
The probability of event A is not actually given by any one label in this diagram, but it is given by the total area inside the circle A, 5/12 + 1/12 = 1/2. The probability of event B is given by the total area inside the circle B, 1/12 + 1/6 = 1/4. And the probability of both events A and B occurring is given by the area of their intersection, 1/12.
But, given that B occurs, what can you say about the probability of event A occurring? Another way of asking this question is: if we pick a random point inside the circle B, what is the probability that it will also lie inside the circle A?
Now we care about the area of the intersection between A and B, but not its absolute value -- what we really care about is the fraction of the total area of B, that is occupied by the intersection of A and B. The total area of B is 1/4. The area of the intersection is 1/12. So to answer the question: What proportion of the area of B, is made up of the intersection of A and B, we divide 1/12 by 1/4 to get 1/3. That is the answer -- given that B occurs (i.e. we pick a random point inside circle B), the probability that event A also occurs (i.e. the probability that the point will also lie inside circle A) is 1/3.
Definition: For two events A and B, the probability of event A, given that event B occurs, is written p(A | B).
Statement 7: For two events A and B, the probability of event A, given that event B occurs,
is given by the formula:
| p(A | B) = |
|
This is the formula that we implicitly used to calculate p(A | B) in the previous example. (Note this formula is true for all types of conditional probabilities, whether the events are independent or not.)
When two events are independent of each other, most of the preceding arguments do not apply, because rather than there being a "maximum possible probability" of A-and-B, or A-or-B, the probability of A-and-B and A-or-B is precisely determined by the probabilities of A and B.
Intuitively, two events A and B are independent if they don't affect each other, e.g. the probability of rain falling tomorrow is independent of the probability of Bill Gates having a ham sandwich for lunch.
Formally, two events A and B are independent of each other if knowing that event A occurs, tells you nothing about the probability that event B occurs, and vice versa. In other words, p(A | B) = p(A) -- which is another way of saying that, given that B occurs, the probability that event A occurs is the same as it would be even if you didn't know that B had occurred.
From this fact, we can derive a formula for probabilities of independent events that you will probably recognize:
Suppose A and B are independent. We want to find p(A and B), the probability of both events occurring. We know
that:
| p(A | B) = |
|
Statement 8: If A and B are independent, then p(A and B) = p(A)*p(B)
Example: If the probability of Bill Gates having a ham sandwich for lunch tomorrow is 1/2, and the probability that it rains tomorrow is 1/5, then the probability of both events occurring is 1/2*1/5 = 1/10 -- because the events are clearly independent of each other.
Note however that if two events are independent, that means that when drawing the Venn diagram of their
probabilities, we are not free to move around the regions A and B to make them overlap as much or
as little as we want them to, because the area of their overlap -- the probability of events A and B
occurring -- is fixed at p(A)*p(B). For example, if events A and B are independent, and p(A) = 1/4 and
p(B) = 1/5, then the probability Venn diagram can only be filled in the following way:
You can prove that each region must be labelled that way, with the following argument:
If the probability of A and B is p(A)*p(B) for independent events, what about the probability of A or B?
Recall from statement 5 that
p(A or B) = p(A) + p(B) - p(A and B)
Since p(A and B) = p(A)*p(B) for
independent events, this means that for two independent events A and B:
p(A or B) = p(A) + p(B) - p(A)*p(B)
|
Note: When solving a given problem, you usually know that two events are independent, either because the problem tells you (e.g. "The probability of Mary coming to the party is independent of the probability of John coming"), or because they are obvious from the circumstances (e.g. the example of rain falling and Bill Gates having a ham sandwich for lunch). But be careful not to assume that two events are independent, and blindly apply the p(A)*p(B) formula, if they are not truly independent of each other. A trick question I like to ask is, "If the probability that it rains on Saturday is 1/5, and the probability that it rains on Sunday is 1/2, what is the probability that it will rain on both days?" There's actually not enough information to answer the question, but the "obvious" answer -- 1/2 * 1/5 = 1/10 -- is wrong, because rain on Saturday and rain on Sunday are classic examples of events that are not independent, since if it rains on Saturday that increases the likelihood that it will rain on Sunday. Even smart people get this wrong! One of my college exams included a question that gave the probability that a voter would like Elizabeth Dole, and the probability that a voter would like George Bush, and asked what was the probability of both events occurring -- however, the question was invalid because the two events are not independent; a voter who likes Dole is more likely than average to like Bush as well. |
Another common type of probability question would be: Suppose you have a bag containing 5 red marbles and 3 blue marbles. You draw one marble from the bag, and then draw another marble without replacing. What is the probability that you will draw 1 red marble followed by 1 blue marble?
To solve this problem, you can use a diagram that is known by many names but which we can call a
"state/probability tree". Begin with your initial state, in which you have 5 red marbles and 3 blue
marbles in the bag:
In this state, the probability of drawing a red marble is 5/8. But note that the act of drawing the red
marble has put you into a different state, in which there are 4 red and 3 blue marbles left. So draw
a path from the initial state to the new state, and mark that path with two labels: the probability of
choosing that path from the previous state, and the action that you took to go down that path:
| Note: The reason that there are now 4 red and 3 blue marbles in the bag is because we chose a marble from the bag without replacing it. If we had put the marble back in the bag after choosing it, then the second state in the state diagram would still show 5 red and 3 blue marbles in the bag. So pay close attention to words like "without replacing" in the statement of the problem! |
Now that there are 4 red and 3 blue marbles left in the bag, ask the question:
given that you have already chosen a red marble, what is the probability of choosing
a blue marble from the bag? Clearly the answer is 3/7, since there are now 7 marbles in the bag and 3
are blue. So draw another path representing that action, which leads to a new state:
Now to answer the original question: What is the probability of reaching that final state, in which you have draw one red marble followed by one blue marble? Call event "A" the event of drawing an initial red marble, and event "B" is the event of drawing a blue marble after you have already drawn a red marble. We want to find p(A and B), the probability that both events occurring.
We know that the probability of drawing an initial red marble is 5/8, so:
p(A) = 5/8
We also know that, given that event A has already occurred, the probability of then drawing a blue
marble (event B) is 3/7, so:
p(B | A) = 3/7.
Now apply the formula for conditional probabilities:
| p(B | A) = |
|
This is a specific example of a general method:
Statement 9: To find the probability of reaching a given state in a state/probability tree, if there is only one sequence of paths that reaches that state, travel the sequence of paths and multiply the probabilities with which the paths are labeled.
Notice that in drawing the state/probability tree for this problem, we did not need to draw the branches corresponding to other choices that we could have taken (for example, starting from the initial state and drawing a blue marble with probability 3/8), because we only cared about a specific sequence of events: drawing a red, and then drawing a blue.
Now, change the question slightly: suppose you want to find out the probability of drawing a red marble and a blue marble, in any order? In other words, what is the probability of drawing a red followed by a blue, OR drawing a blue followed by a red?
We've already labeled the states that correspond to the probability of drawing a red followed by a blue:
So, fill in some more paths in the tree. The probability of drawing a blue marble initially is 3/8, which
would put you in a state of having 5 red and 2 blue marbles left in the bag:
And given that you are in the state of having 5 red and 2 blue marbles in the bag, the probability of
then drawing a red marble is 5/7:
Using the same technique, we conclude that the probability of reaching that state via the red-blue path
is also 15/56:
So, say that event A is the event that you draw a red followed by a blue, and event B is the event that you draw a blue followed by a red. Clearly, these events are mutually exclusive -- they cannot both happen. Recall Statement 6: If A and B are mutually exclusive events, then the probability of event A or event B occurring is p(A) + p(B). So the probability that you will draw a red followed by a blue, or a blue followed by a red, is 15/56 + 15/56 = 15/28. In general:
Statement 10: To find the probability that you will reach some set of mutually exclusive states in a state/probability diagram, find the probabilities of reaching each individual state, and add them.
|
Note: you can also solve this problem using combinatorial methods, as explained in the combinatorics tutorial. Given that there are 5 red and 3 blue marbles in the bag, if you choose two at random, what is the probability that exactly 1 will be red and exactly 1 will be blue? Since we don't care about the order in which the marbles are chosen, that means the number of pairs of marbles that can be chosen from the bag is 8-choose-2, or 28. Out of those pairs, how many involve exactly 1 red and exactly 1 blue? There are 5 possible choices for the 1 red marble, and 3 possible choices for the 1 blue marble, so there are 5*3 = 15 possible red-blue pairs. So with 28 possible pairs, and 15 possible pairs that meet the condition, the probability that your randomly chosen pair will consist of 1 red and 1 blue marble, is 15/28. |
Practice problem: Starting again with a bag containing 5 red and 3 blue marbles, find the probability of drawing 2 red marbles without replacing. First find the answer using a state/probability diagram using the methods above. Then find the answer using combinatorial methods: how many possible pairs are there, and how many possible pairs meet the criteria? The answer in both cases should be 5/14.
Practice problem: Start again with a bag containing 5 red and 3 blue marbles, and draw two marbles, except this time the rules have changed: If you draw a red marble, you put it back in the bag, but if you draw a blue marble, you keep it out. Now what is the probability of drawing a red marble and a blue marble, in any order?
Hint: Use a state/probability tree to solve the problem using the methods above, except this time, when your first choice is the red marble, the state that leads to still has 5 red and 3 blue marbles in the bag. You should get a 15/64 chance of getting a red followed by a blue, and a 15/56 chance of getting a blue followed by a red, so the answer should equal 15/56 + 15/64 = 225/448.
Note: in this case there is unfortunately no easy way to check your answer using combinatorial methods, because it's possible for the same red marble to be "chosen twice".