Probability and Distributions

A distribution answers several questions

Choose the object by what it answers
Object	Input and output	Useful question
Probability mass function (PMF)	Value → nonnegative mass	How likely is this discrete outcome?
Probability density function (PDF)	Value → nonnegative density	How much probability is near this value?
Cumulative distribution function (CDF)	Threshold → probability	How much lies at or below x?
Joint distribution	Tuple → mass/density	How do outcomes occur together?
Characteristic function	Frequency → complex average	How do sums or distributional shapes behave?

A random variable maps outcomes to numbers. Its distribution describes the probabilities of those numbers, not one realized sample. The same distribution can have several equivalent descriptions, but not every distribution has a density.

Conditioning changes the reference population

\begin{aligned}P(A\mid B)&=\frac{P(A\cap B)}{P(B)},\quad P(B)>0\\P(A\cap B)&=P(A\mid B)P(B)\end{aligned}

Conditioning restricts attention to outcomes in B, then renormalizes.

P(A|B) and P(B|A) usually differ because their denominators differ. Independence means P(A∩B)=P(A)P(B). For positive P(B), it then gives P(A|B)=P(A). Mutually exclusive positive-probability events are not independent: observing one rules out the other.

A zero-probability condition cannot be handled by this elementary ratio. Conditioning on an exact continuous value needs the appropriate conditional-distribution construction.

Combine disjoint cases before reversing them

\begin{aligned}P(A)&=\sum_i P(A\mid B_i)P(B_i)\\P(B_j\mid A)&=\frac{P(A\mid B_j)P(B_j)}{\sum_iP(A\mid B_i)P(B_i)}\end{aligned}

The Bᵢ form a mutually exclusive, exhaustive partition. Terms with zero case probability contribute zero; their undefined elementary conditional ratio is not evaluated.

The first identity follows by writing A as the disjoint union of A∩Bᵢ. The second is Bayes' rule with that total as its normalizer. Conditional rates are weighted by case frequencies; an unweighted average is valid only for equal case probabilities.

Consider a hypothetical diagnostic test: prevalence 1%, sensitivity 99%, false-positive rate 2%. These are specified toy rates, not medical advice or a claim about a real test. Among 10,000 expected cases, 100 have the condition and 9,900 do not.

Expected counts make the denominator visible
Branch	Population	Positive rate	Expected positives
Condition	100	.99	99
No condition	9,900	.02	198
Total	10,000	weighted total	297

\begin{aligned}P(+)&=0.99(0.01)+0.02(0.99)=0.0297\\P(D\mid+)&=\frac{99}{297}=\frac13\end{aligned}

D denotes the condition and + a positive result. Sensitivity 99% is not a 99% posterior. Most positive results in this toy population are false positives because the condition is rare.

Expected counts need not be integer realized counts in other examples. They are a convenient scale for the same probability calculation, not a guarantee about one sampled population.

Read a joint table in two directions

\begin{aligned}p_{ij}&=P(X=x_i,Y=y_j)\\P(X=x_i)&=\sum_j p_{ij}\\P(Y=y_j)&=\sum_i p_{ij}\\P(Y=y_j\mid X=x_i)&=\frac{p_{ij}}{\sum_k p_{ik}}\end{aligned}

Summing eliminates a variable. Conditioning selects a positive-mass row and renormalizes it.

Check nonnegative entries and a total of one before using a table. Each marginal must also sum to one. The same law of total probability appears here: weighted conditional rows reconstruct the marginal of Y.

\begin{aligned}\mathbb E[X]&=\sum_i x_iP(X=x_i)\\\mathbb E[Y]&=\sum_j y_jP(Y=y_j)\\\mathbb E[XY]&=\sum_{i,j}x_i y_j p_{ij}\\\operatorname{Cov}(X,Y)&=\mathbb E[(X-\mathbb E[X])(Y-\mathbb E[Y])]\\&=\mathbb E[XY]-\mathbb E[X]\mathbb E[Y]\end{aligned}

Assume the required expectations exist. Expand the centered product to obtain the shortcut.

For binary X and Y, take $p_{00}=p_{11}=0.4$ and $p_{01}=p_{10}=0.1$ . Both means are .5, $\mathbb E[XY]=0.4$ , and $\operatorname{Cov}(X,Y)=0.4-0.25=0.15$ . The following family changes this association while keeping both marginals fixed.

Same marginals, different joint distributions

X and Y are each Bernoulli(½). Moving c changes only how their outcomes pair. Table rows are X; columns are Y.

Covariance 0.15

Joint probability mass
	Y=0	Y=1	P(X)
X=0	0.40	0.10	0.50
X=1	0.10	0.40	0.50
P(Y)	0.50	0.50	1.00

E[XY] 0.40Correlation 0.60P(Y=1|X=1) 0.80

More mass is on equal outcomes: positive covariance.

Exact four-cell family. At c=0 it is independent; this special binary family does not establish the reverse implication for all distributions.

Zero covariance is not independence

Positive covariance indicates a positive signed linear association; negative covariance indicates the reverse. Zero covariance means the centered product has zero expectation, not that every form of dependence has vanished. Correlation divides covariance by the two standard deviations and is undefined if either is zero.

Take X uniform on {−1,0,1} and $Y=X^2$ . Then $\mathbb E[X]=0,\quad\mathbb E[Y]=\frac23$ and $\mathbb E[XY]=\mathbb E[X^3]=0$ . Covariance is zero, yet Y is determined by X. The dependence is nonlinear.

Independence implies zero covariance when second moments are finite. The reverse is true under additional assumptions, such as jointly Gaussian variables. It is false in general.

A covariance matrix records pairwise second-order structure. Identity covariance means unit variances and zero cross-covariances, not necessarily independent or Gaussian coordinates. Sketched Isotropic Gaussian Regularization (SIGReg) compares projected distributions with a Gaussian target. Matching covariance alone cannot establish that target; we return to this distinction after introducing characteristic functions.

Accumulate probability across a threshold

\begin{aligned}F(x)&=P(X\le x)\\P(a<X\le b)&=F(b)-F(a),\quad a\le b\end{aligned}

Endpoint inclusion matters at atoms. P(X=x)=F(x)−F(x−), where F(x−) is the left limit.

For example, F(3)=0.8 means that 80% of the mass lies at or below 3.

A CDF is nondecreasing, right-continuous, takes values in [0,1], and tends to zero/one at the two infinities. Right-continuity fixes how jumps include their own value. It fully determines the distribution, including discrete, continuous and mixed cases.

Fair die: cumulative steps
Threshold	F(x)
x<1	0
1≤x<2	1/6
2≤x<3	2/6
3≤x<4	3/6
4≤x<5	4/6
5≤x<6	5/6
x≥6	1

For Uniform(0,1), F(x)=0 below zero, x between zero and one, and 1 above one. Thus P(.2<X≤.7)=.7−.2=.5.

\begin{aligned}\text{Discrete: }F(x)&=\sum_{t\le x}P(X=t)\\\text{With density }f:\;F(x)&=\int_{-\infty}^x f(t)\,dt\end{aligned}

For an absolutely continuous distribution, f=F′ almost everywhere. Differentiability alone at selected points does not ensure that the full CDF is recovered by integrating those derivatives.

A PDF can exceed one; interval area, not height alone, is probability. For a distribution with a density, exact points have zero probability. An atom is different: its CDF jumps. A general continuous CDF can also be singular, so “continuous” is not automatically “has a PDF.”

Quantiles and inverse-transform sampling

\begin{aligned}Q(u)&=\inf\{x:F(x)\ge u\},\quad0<u<1\\U&\sim\operatorname{Uniform}(0,1)\\&\Longrightarrow Q(U)\text{ has CDF }F\end{aligned}

This generalized inverse works for jumps and flat regions. It does not require a strictly increasing CDF.

A fair die has $Q(u)=\lceil6u\rceil,\quad0<u<1$ . Intervals of u with width 1/6 select each face. The cumulative mass at a discrete quantile can exceed the requested level because the CDF jumps there.

The 95th percentile is the lower threshold whose cumulative mass reaches .95 under this convention. It need not have exactly 95% below it. Sample-percentile libraries may interpolate using different conventions; record the chosen one.

Uniform endpoints have probability zero in the continuous idealization, but software can produce boundary values. For the finite-support helper used here, zero selects the first positive-mass value and one selects the last. Zero-mass entries are never returned. Tiny cumulative rounding errors use the last positive-mass value, not an unused trailing category.

Observed frequencies are an estimate

F_n(x)=\frac1n\sum_{j=1}^n\mathbf1\{X_j\le x\}

Count observations at or below the threshold, then divide by sample size.

For [1,2,2,5], $F_n(2)=\frac34$ . The repeated value creates a jump of 2/4 at 2. Ties should be counted, not silently deduplicated.

Empirical CDFs support percentile lookup, interval counts and distribution comparisons. The Kolmogorov–Smirnov statistic uses the largest CDF discrepancy. Its usual calibration assumes particular sampling and null conditions; fitted parameters or ties can require different calibration.

Data dependence, selection bias and a small sample can make an empirical view misleading. A smooth plot is not evidence that the sample represents the population.

The same distribution in frequency space

A CDF answers threshold questions directly. For a sum of random variables, combining many thresholds becomes cumbersome. Frequency space offers another complete representation: rotate each value into a unit vector, then average.

\begin{aligned}\varphi_X(t)&=\mathbb E[e^{itX}]\\&=\mathbb E[\cos(tX)]+i\mathbb E[\sin(tX)]\end{aligned}

t is a frequency and

i^2=-1

. Every outcome contributes a unit-circle vector; average with its probability.

The characteristic function always exists because the complex exponential has magnitude one. In particular φ(0)=1, |φ(t)|≤1 and φ(−t)=conjugate(φ(t)). Negative real parts or imaginary components are not negative probabilities; they are averages of oscillating coordinates.

Exact examples
Distribution	Characteristic function
Constant X=c	$e^{itc}$
Bernoulli(p)	$1-p+pe^{it}$
Standard normal	$e^{-t^2/2}$
Normal(μ,σ²)	$e^{it\mu-\sigma^2t^2/2}$

The full function uniquely determines the distribution. Matching a few frequencies does not. Small t emphasizes slower oscillation and, when moments exist, its local derivatives encode them. Larger |t| can reveal fine structure, but “global versus local” is an intuition, not a universal recovery guarantee.

Two views of one Bernoulli distribution

Dashed burgundy vectors represent outcomes 0 and 1 on the unit circle. Solid blue is their probability-weighted mean φ(t), not another sample.

P(X=1) 0.50Frequency t 0.80

Real 0.8484Imaginary 0.3587Magnitude 0.9211

The mean stays inside the unit disk. Unequal phases can cancel even though probabilities are nonnegative.

CDF threshold x 0.50

F(x)=P(X≤x): 0.50. Only outcome 0 is included.

Playback sweeps frequency only. Threshold and frequency are independent views. This integer-support CF is periodic; one selected frequency cannot identify an arbitrary distribution.

Independent sums become products

\begin{aligned}\varphi_{X+Y}(t)&=\mathbb E[e^{itX}e^{itY}]\\&=\mathbb E[e^{itX}]\mathbb E[e^{itY}]\\&=\varphi_X(t)\varphi_Y(t)\end{aligned}

The factorization uses independence. Without it, the joint characteristic function is needed.

Convolution in value space becomes multiplication in frequency space. For independent normals, multiplying the two exponential forms adds their means and variances. For n independent Bernoulli(p) trials, $(1-p+pe^{it})^n$ describes their binomial sum.

An affine transformation also has a simple form: $\varphi_{aX+b}(t)=e^{itb}\varphi_X(at)$ . Here a and b are fixed real constants. Shifting adds phase; scaling changes the frequency argument. This makes the representation convenient for algebra, not only for plots.

Moments require existence conditions

\begin{aligned}\mathbb E[|X|^k]<\infty&\Longrightarrow\varphi_X^{(k)}(0)=i^k\mathbb E[X^k]\\\mathbb E[X]&=\frac{\varphi_X'(0)}{i}\\\mathbb E[X^2]&=-\varphi_X''(0)\end{aligned}

Finite absolute moments justify exchanging differentiation and expectation. Do not infer all moments from the existence of φ itself.

Variance is E[X²]−E[X]² when these moments exist. A characteristic function can exist even when the mean or variance does not. Moment-generating functions E[eᵗˣ] lack the bounded complex-exponential argument and may fail to exist around zero.

For a vector X, $\varphi_X(t)=\mathbb E[e^{i t^\top X}]$ uses a vector frequency of the same dimension. The dot product gives the scalar phase. Individual projected distributions examine directions, while a covariance matrix captures only second-order structure.

Compare distributions, not only their covariance

\widehat\varphi_{X,n}(t)=\frac1n\sum_{j=1}^n e^{itX_j}

The empirical characteristic function replaces an expectation with a finite average.

The full characteristic function contains more than means and covariance. We can therefore compare distributional shape, including departures that those summaries miss.

For a normality comparison, standardize observations $Y_j=\frac{X_j-\bar X}{S}$ with positive scale S. Recompute the empirical function from Y rather than X: replace Xⱼ with Yⱼ in the displayed average. Compare this standardized function with the standard-normal target. A constant sample has S=0 and requires a separate policy.

T_n=n\int_{-\infty}^{\infty}\left|\widehat\varphi_{Y,n}(t)-e^{-t^2/2}\right|^2w(t)\,dt

The Epps–Pulley family uses a nonnegative frequency weight. Larger mismatch gives a larger statistic, not automatically a calibrated probability.

The integrand is the squared complex distance at one frequency. Weighting and integrating combines these mismatches. Gaussian weights emphasize a frequency range and make the integral reducible to finite sums through Gaussian Fourier identities. Weight bandwidth affects sensitivity; high-frequency empirical oscillations should not be interpreted as pure population structure.

Mean/scale estimation, sample size, standardization convention and the null model affect test calibration. Do not attach a p-value without the corresponding procedure. Failure to reject is not proof of normality.

Distributional regularizers can borrow this comparison idea without being a classical hypothesis test. Projected Gaussian regularization → applies frequency-space comparisons to projected embeddings. An optimization penalty and a calibrated statistical test have different uses.

Choose the representation for the question

A small probability workflow
Question	First operation	Check
Different scenarios	Partition and weight conditional rates	Disjoint and exhaustive cases
Reverse a condition	Bayes numerator / total denominator	Base rates and positive denominator
Joint variability	Marginals, E[XY], centered product	Finite moments; units; not independence
Intervals or percentiles	CDF difference or generalized inverse	Open/closed endpoints and quantile convention
Sums or affine changes	Characteristic multiplication / argument transform	Independence and moment conditions
Distributional mismatch	CDF or frequency comparison	Sampling and statistical calibration

Expectation, counting, stopping rules and Poisson arrival models are grouped in the Mathematical Problem Solving →. Linearity of expectation does not require independence; a Poisson arrival model does require its process assumptions. Those distinctions belong with the problem-solving workflow, not another isolated probability page.

Information Theory → then asks how much coding or predictive uncertainty a distribution contains. Probability defines the distribution first; entropy is a functional of it, not a replacement for the full distribution.

Sources and connected owners

Characteristic functions: definition and conditions ↗ · Epps and Pulley (1983) ↗