Conditional probability and rules of probability Knowledge-Based Study

Study notes

1. What is Conditional Probability?

   Conditional probability measures the probability of an event occurring given that another event has already occurred. This condition effectively restricts or reduces the original sample space to the outcomes contained in the conditioning event.

   For two events A and B in a sample space S, with $P(B) > 0$, the conditional probability of A given B is written as $P(A|B)$ and defined by:

   $$P(A|B) = \frac{P(A \cap B)}{P(B)}$$

   Here, $P(A \cap B)$ represents the joint probability that both events A and B occur, while $P(B)$ represents the probability that the conditioning event B occurs.

   The General Multiplication Rule

   Rearranging the conditional probability formula yields the general multiplication rule, which allows us to find the probability of the intersection of two events:

   $$P(A \cap B) = P(B) \times P(A|B)$$

   Similarly, when conditioning on event A (assuming $P(A) > 0$):

   $$P(A \cap B) = P(A) \times P(B|A)$$

   Example: Sampling Without Replacement

   Consider drawing two cards consecutively without replacement from a standard deck of 52 playing cards. Let A be the event that the first card is an Ace, and let B be the event that the second card is an Ace.

   • $P(A) = \frac{4}{52}$
   • $P(B|A) = \frac{3}{51}$ (since one Ace and one card are removed)
   • $P(A \cap B) = \frac{4}{52} \times \frac{3}{51} = \frac{12}{2652} = \frac{1}{221}$

   Testing for Independence

   Two events A and B are said to be independent if the occurrence of one does not affect the probability of the other. In mathematical terms, A and B are independent if and only if:

   $$P(A|B) = P(A) \quad \text{or equivalently} \quad P(B|A) = P(B)$$

   Substituting this relationship into the general multiplication rule gives the special multiplication rule for independent events:

   $$P(A \cap B) = P(A) \times P(B)$$

   Common Misconception

   Do not confuse independent events with mutually exclusive (disjoint) events. Disjoint events satisfy $P(A \cap B) = 0$. If two events with non-zero probabilities are mutually exclusive, they cannot be independent because knowing that one has occurred guarantees that the other cannot occur: $P(A|B) = 0 \neq P(A)$.

   Partitioning the Sample Space

   Let $B_1, B_2, \dots, B_k$ form a partition of the sample space S. This means the sets are mutually exclusive ($B_i \cap B_j = \emptyset$ for all $i \neq j$) and exhaustive (their union is S), with $P(B_i) > 0$ for each i.

   For any event A defined on the same sample space, the Law of Total Probability states:

   $$P(A) = \sum_{i=1}^{k} P(A \cap B_i) = \sum_{i=1}^{k} P(B_i) \times P(A|B_i)$$

   For a binary partition where B and its complement $B^c$ divide the space, this simplifies to:

   $$P(A) = P(B)P(A|B) + P(B^c)P(A|B^c)$$

   Inverting Conditional Probabilities

   Bayes' Theorem provides a method to reverse conditional probabilities, calculating $P(B|A)$ from known values of $P(A|B)$. By combining the definition of conditional probability with the Law of Total Probability, we obtain:

   $$P(B|A) = \frac{P(A \cap B)}{P(A)} = \frac{P(B)P(A|B)}{P(B)P(A|B) + P(B^c)P(A|B^c)}$$

   More generally, for a partition $B_1, B_2, \dots, B_k$, Bayes' Theorem states:

   $$P(B_j|A) = \frac{P(B_j)P(A|B_j)}{\sum_{i=1}^{k} P(B_i)P(A|B_i)}$$

   In this formulation, $P(B_j)$ is referred to as the prior probability, and $P(B_j|A)$ is referred to as the posterior probability updated by the evidence of event A.