Mathematics · Intermediate · 40 min · Chapter 10 of 28

Probability and statistics

Probability is the mathematics of uncertainty. Statistics is the reverse: data in hand, what can you say about the process that made it?

Read first: Algebra, Functions, exponentials, and logarithms

The idea

An outcome is a possible result. An event is a set of outcomes. Probability assigns each event a number between 0 and 1 so that the whole sample space has probability 1 and disjoint events add. If outcomes are equally likely, probability is a count: favorable over total. That model is a special case, not the definition. A biased coin is still a perfectly good probability space.

Conditional probability P(A | B) = P(A and B) / P(B) is the probability of A once you know B happened. Independence means learning B changes nothing: P(A | B) = P(A), equivalently P(A and B) = P(A)P(B). Bayes’ theorem runs the conditioning backwards, from the probability of the evidence given a cause to the probability of the cause given the evidence. It is arithmetic, not a philosophy, and it is the reason a positive test for a rare condition can still leave the condition unlikely.

A random variable is a number attached to each outcome. Its expectation E[X] is the probability-weighted average of its values. For an indicator of an event, expectation equals the probability. Variance is E[(X − μ)²], the expected squared distance from the mean; the standard deviation is its square root, back in the original units. The sample mean of many independent draws concentrates around the true mean, and the distribution of that sample mean approaches a normal curve as the sample grows. That is the central limit theorem, and it is why the bell curve appears even when the underlying measurements are not themselves bell-shaped.

Hold onto this

A probability is a number from 0 to 1 assigned to events, additive over events that cannot happen together.
Conditional probability updates a belief when you learn that another event occurred.
Expectation is the long-run average. Variance measures spread around it.
The central limit theorem is why averages so often look bell-shaped.

Formulas

Conditional probability

P (A ∣ B) = \frac{P ( A \cap B )}{P ( B )}

The probability of A given that B occurred. Requires P(B) > 0.

A ∩ B: Both A and B

Bayes’ theorem

P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B )}

Swap cause and evidence. The denominator is the total probability of the evidence.

P(A): Prior probability of the cause
P(B | A): Likelihood of the evidence if the cause is true

Binomial probability

P (X = k) = (k n) p^{k} (1 - p)^{n - k}

Probability of k successes in n independent trials, each with success probability p.

C(n, k): The binomial coefficient n! / (k!(n − k)!)

Expectation and variance

μ = E [X], Var (X) = E [(X - μ)^{2}]

Mean, and expected squared deviation from the mean.

μ: Population mean

Standard normal density

ϕ (z) = \frac{1}{2 π} e^{- z^{2} /2}

The bell curve with mean 0 and standard deviation 1. General normals are shifts and scales of this one.

z: A standardized value: (x − μ) / σ

Worked examples

1. A rare disease and a good test

A condition affects 1% of people. A test is positive for 99% of people who have it, and for 5% of people who do not. You test positive. What is the chance you have it?

P(D) = 0.01, P(+ | D) = 0.99, P(+ | not D) = 0.05.
Total P(+) = 0.99(0.01) + 0.05(0.99) = 0.0099 + 0.0495 = 0.0594.
P(D | +) = 0.0099 / 0.0594 ≈ 0.167.

About 16.7%. Most positive tests are false, because the condition is rare and the false-positive rate, though small, applies to almost everyone.

2. A fair binomial

You flip a fair coin 4 times. What is the probability of exactly 3 heads?

n = 4, k = 3, p = 1/2.
C(4, 3) = 4.
4 × (1/2)³ × (1/2)¹ = 4/16 = 1/4.

1/4. The four sequences are HHHT, HHTH, HTHH, THHH.

Try these

A die is rolled. Let A be “even” and B be “greater than 4.” Are A and B independent?

Expected value of one roll of a fair six-sided die?

Common mistakes

P(A | B) is not P(B | A). The disease example exists to make that difference numerical.
Independent is not the same as disjoint. Disjoint events with positive probability are dependent: if one happens, the other cannot.
A small p-value is not the probability that the hypothesis is true. It is the probability of data at least this extreme if the hypothesis is true.