Conditional Probability

The concept of conditioning in Statistics is of great importance in many fields of applications, such as, information theory, finance, random signal processing etc. In most of the cases, when the events are random and dependent and we want to specify the probability of occurrence of one given the other event, we use the concept of conditional probability.

Consider there are red and black balls in a box. We are picking the balls at random form the box and we do not put back in again, then the two events i.e. picking a red ball and picking a black ball at random are dependent. When we pick the first ball, the probability does not depend on the previous event as it is the initial pick and probability of picking any one ball is equally likely or in other words, they are uniformly distributed.

But when we pick the  second ball, there is now one ball less in the box and the probability of occurrence of the second event is dependent on the first event. In other words, we can condition the occurrence of event II on the known result of event one. We can now determine "what is the probability of occurrence of event II given the event I has already occurred?". This approach of determining the probability of occurrence of any event is called the conditional probability.

Truly speaking, the concept of conditioning is the heart of probability theory and we can use this concept to define the law of total probability based on the Bayes' rule.

Let us consider a combined experiment in which two events $A$ and $B$ occur with joint probabilities $P(A,B)$. Joint probability is the probability of occurring both the events. The probability of occurrence of  $B$ given $A$  is the conditional probability and is represented as $P(B|A)$, which is expressed in terms of the joint probability as :

                               $P(B|A)=\frac{P(A,B)}{P(A)}$.                     

 Similarly, the probability of occurrence of A conditioned on B is mathematically expressed as

                              $P(A|B)=\frac{P(A,B)}{P(B)}$.             

In each case $P(A), P(B) > 0$, because the value of probability always lies between 0 and 1 (axiomatic definition). Combining above two conditions, we can write 

                             $P(B|A)P(A)=P(A|B)P(B)$ 

                            $\Rightarrow P(B|A)=\frac{P(A|B)P(B)}{P(A)}.$                       

This is called the famous Bayes' Rule in statistics.

Say $A=\{1,2,3\}$ and $B=\{3,4,5\}$ are the two events occurred when rolling a die. It means $P(A)=3/6=1/2$ and $P(B)=3/6=1/2$, they are independent events and $P(A,B)=1/6$

                         so, $P(B|A)=\frac{P(A,B)}{P(A)}=\frac{1/6}{1/2}=\frac{1}{3}$.

and for mutually exclusive events, $P(A,B)=\phi=0$. So conditional probability is zero in such case.

If $P(A,B)=P(A)$ i.e A is a subset of B, then $P(B|A)=P(A)/P(A)=1.$

and if $P(A,B)=P(B)$ i.e. B is a subset of A, then $P(B|A)=P(B)/P(A).$

Conditional probability is also useful in checking the statistical independence. If the occurrence of A does not depend on the occurrence of B then

              $P(A|B)=P(A)$ 

       and
           $P(A,B)= P(A)P(B)$

Law of total probability states that if there are $n$ mutually exclusive events represented as $E_1, E_2 \cdots E_n$, the sum of the probabilities of which sum to 1 and $F$ is some other arbitrary event, then the probability of $F$ is expressed in terms of the conditional probability as:
$P(F)=P(F|E_1)P(E_1)+ P(F|E_2)P(E_2) + \cdots + P(F|E_n)P(E_n) $.