% thin space, limits underneath in displays % thin space, limits underneath in displays }}
Introduction and data used
We illustrate in this post the notion of ROC curves and Precision-Recall curves using both simple distributions (like Gaussian or uniform) but also data from a head-to-head benchmark evaluation of five SARS-CoV-2 serological assays (Ainsworth et al. 2020). We focus only on binary classification in this post.
For the empirical part, the underlying study compared the Abbott, DiaSorin, Oxford, Roche, and Siemens assays using a common set of well-characterized samples. We follow Ainsworth et al. (2020) and have used a dataset that contains two groups of samples:
976 pre-pandemic samples, used as the known-negative reference group;
536 laboratory-confirmed SARS-CoV-2-positive samples, collected at least 20 days after symptom onset.
These groups contain a total of
\[
n = 976 + 536 = 1512
\]
observations. The proportion of positive samples in this dataset is therefore
An important distinction: this proportion describes the composition of the dataset, not necessarily the prevalence of SARS-CoV-2 infection in the population where a diagnostic test would be used. The sample composition may reflect how the study was designed and how samples were selected. This distinction will become important when interpreting precision–recall curves and average precision, which depend on the prevalence of positive cases.
The dataset (supplementary_dataset2.csv) is available here. The supplementary material of the associated paper describes the preprocessing used in this analysis.
Diagnostic testing as hypothesis testing
A diagnostic test can be framed as a classical statistical hypothesis test. In this medical context, we consider the null hypothesis \[
\mathcal{H}_0: \text{``you are healthy''}
\] against the alternative: \[
\mathcal{H}_1: \text{``you are sick''}.
\]
The presumption of health plays the same role as the presumption of innocence in a courtroom: we only reject \(\mathcal{H}_0\) (i.e., declare someone positive) when the evidence (here the assay result) is strong enough. Let \(Y \in \{0,1\}\) denote the true health status, with \(Y=0\) under \(\mathcal{H}_0\) (healthy) and \(Y=1\) under \(\mathcal{H}_1\) (sick). In practice, declaring someone positive means the measured antibody level, modeled by a random variable \(X\), must cross an assay-specific diagnostic threshold (cut-off) \(t\). The resulting decision rule, our estimate of \(Y\) at threshold \(t\), is written \(\hat{Y}_t\):
\[
\hat{Y}_t(X) = \mathbb{1}\{X \ge t\}=
\begin{cases}
1, & X \ge t,\\
0, & X < t .
\end{cases}
\] When no ambiguity is possible we simply write \(\hat{Y}_t\).
Throughout this post, \(\hat{Y}_t\) denotes this rule, except in the symmetry sections where the reversed rule \(\mathbb{1}\{X<t\}\) is considered explicitly.
It is convenient to isolate the two conditional laws of \(X\) given the health status. Under \(\mathcal H_0\), \(X\) follows \(\mathcal{L}(X\mid Y=0)\); under \(\mathcal H_1\), it follows \(\mathcal{L}(X\mid Y=1)\). We write \[
\begin{align*}
X_0 & \sim \mathcal{L}(X\mid Y=0), \\
X_1 & \sim \mathcal{L}(X\mid Y=1)
\end{align*}
\] for two independent random variables with these laws. This turns any probability conditional on \(Y\) into an ordinary, unconditional probability: for every threshold \(t\), \[
\begin{align*}
\mathbb{P}(X \ge t \mid Y=0)
& = \mathbb{P}(X_0\ge t),
\\
\mathbb{P}(X \ge t \mid Y=1)
& = \mathbb{P}(X_1\ge t),
\end{align*}
\] and we use the right-hand sides from here on, since they involve a single random variable rather than a conditional statement.
As in any hypothesis test, two types of errors can occur: rejecting \(\mathcal{H}_0\) when it is in fact true (Type I error), or failing to reject it when it is in fact false (Type II error). In terms of \(X_0\) and \(X_1\):
In words: a Type I error is a false positive (the test flags a healthy person), while a Type II error is a false negative (the test misses a sick person). The threshold \(t\) controls the trade-off between the two: lowering \(t\) catches more true cases but triggers more false alarms, and vice versa. ROC and precision/recall curves, introduced below, are the tools used to visualize this trade-off as \(t\) varies.
Naming and key quantities
The dataset consists of \(n\) observations, denoted \(\{(Y_i, X_i)\}_{i=1,\dots,n}\), where \(Y_i\) represents the true health status of individual \(i\).
Naming follows the classical terminology (see e.g., the Wikipedia article on ROC curves). Given a fixed threshold, each sample falls into exactly one of the four categories below:
False Positive (Type I error): healthy but wrongly declared positive \[\displaystyle\mathrm{FP} = \sum_{i=1}^n \mathbb{1}\{Y_i=0,\ \hat{Y}_{t,i}=1\}\]
False Negative (Type II error): sick but wrongly declared negative \[\displaystyle\mathrm{FN} = \sum_{i=1}^n \mathbb{1}\{Y_i=1,\ \hat{Y}_{t,i}=0\}\]
From the observed true labels, we define the two class counts:
\(n_1=\mathrm{P} = \displaystyle\sum_{i=1}^n \mathbb{1}\{Y_i = 1\} = \mathrm{TP} + \mathrm{FN}\): number of positive (sick) individuals in the sample,
\(n_0=\mathrm{N} = \displaystyle\sum_{i=1}^n \mathbb{1}\{Y_i = 0\} = \mathrm{TN} + \mathrm{FP}\): number of negative (healthy) individuals in the sample,
and, similarly, from the predicted status \(\hat{Y}_t\):
\(\hat{\mathrm{P}}_t = \sum_{i=1}^n \mathbb{1}\{\hat{Y}_{t,i} = 1\} = \mathrm{TP} + \mathrm{FP}\): total number of individuals predicted sick at threshold \(t\) (predicted positive),
\(\hat{\mathrm{N}}_t = \sum_{i=1}^n \mathbb{1}\{\hat{Y}_{t,i} = 0\} = \mathrm{TN} + \mathrm{FN}\): total number of individuals predicted healthy at threshold \(t\) (predicted negative).
Note: we will reuse the same color code below, with red for \(\mathrm{N}\), blue for \(\mathrm{P}\), crosses for a negative test result, and dots for a positive test result.
We also define the following rates, used throughout this post. Let \(\pi_1\triangleq\mathbb{P}(Y=1)\) and \(\pi_0\triangleq\mathbb{P}(Y=0)=1-\pi_1\) denote the class probabilities; we refer to \(\pi_1\) as the prevalence of the disease in the population.
The true rates. These are probabilities of the underlying random variables (population quantities, not counts):
\(\mathrm{TPR}(t) \triangleq \mathbb{P}\!\left(\hat{Y}_t=1 \mid Y=1\right)\) (True Positive Rate, a.k.a. sensitivity or recall): among sick people, the proportion correctly detected. This is also \(\mathbb{P}(X_1\ge t)\).
\(\mathrm{TNR}(t) \triangleq \mathbb{P}\!\left(\hat{Y}_t=0 \mid Y=0\right)\) (True Negative Rate, a.k.a. specificity): among healthy people, the proportion correctly cleared. This is also \(\mathbb{P}(X_0< t)\).
\(\mathrm{FPR}(t) \triangleq \mathbb{P}\!\left(\hat{Y}_t=1 \mid Y=0\right)\) (False Positive Rate): among healthy people, the proportion wrongly flagged, exactly the Type I error probability. This is also \(\mathbb{P}(X_0\ge t)\).
\(\mathrm{Precision}(t) \triangleq \mathbb{P}\!\left(Y=1 \mid \hat{Y}_t=1\right)\): among people declared positive, the proportion actually sick.
The first three depend only on the conditional laws \(X_0,X_1\). Precision does not: it also depends on the class probabilities \(\pi_0,\pi_1\), and so, unlike the other three, cannot be written purely in terms of \(X_0\) and \(X_1\) (its exact form is given in Theorem 11 below).
Estimators. In practice \(t\), \(\pi_1\), and the laws of \(X_0,X_1\) are unknown, and each rate above is estimated from a labeled sample \((X_i,Y_i)_{i=1}^n\). With the notation defined earlier, the naive (plug-in) estimator of each rate simply replaces the expectation defining it by the corresponding empirical mean over the sample: \[
\begin{align*}
\widehat{\mathrm{TPR}}(t) & =\frac{\mathrm{TP}(t)}{\mathrm{P}}, \\
\widehat{\mathrm{TNR}}(t) & =\frac{\mathrm{TN}(t)}{\mathrm{N}}, \\
\widehat{\mathrm{FPR}}(t) & =\frac{\mathrm{FP}(t)}{\mathrm{N}}, \\
\widehat{\mathrm{Precision}}(t) & =\frac{\mathrm{TP}(t)}{\hat{\mathrm{P}}_t}.\\
\end{align*}
\]
The ROC curve
The receiver operating characteristic (ROC) curve comes from signal detection theory, which emerged from radar engineering during World War II and was formalised by Peterson, Birdsall, and Fox (1954). The problem was to decide, from a noisy measurement, whether a target was present or only background noise. A receiver declares “signal” when the measurement exceeds a threshold, and the ROC curve records how the detection rate and the false-alarm rate move together as that threshold varies. This is exactly the trade-off between catching true threats and raising false alarms. In the language of this post, it is the curve of the power of a threshold test against its Type I error.
Definition 1 (ROC curve) For a threshold \(t\in\mathbb{R}\), the ROC operating point is the pair of false and true positive rates,
Step 1: limit as \(t\to+\infty\). By definition, \[
\mathrm{FPR}(t) = \mathbb{P}(X_0\ge t), \quad \mathrm{TPR}(t) = \mathbb{P}(X_1\ge t).
\] Since \(X_0\) and \(X_1\) are almost surely finite, \(\mathbb{P}(X_y\ge t)\to0\) as \(t\to+\infty\) for \(y\in\{0,1\}\), giving \[
\boxed{\lim_{t\to+\infty} \bigl(\mathrm{FPR}(t),\mathrm{TPR}(t)\bigr) = (0,0).}
\]
Step 2: limit as \(t\to-\infty\). Similarly, \(\mathbb{P}(X_y\ge t)\to1\) as \(t\to-\infty\) for \(y\in\{0,1\}\), giving \[
\boxed{\lim_{t\to-\infty} \bigl(\mathrm{FPR}(t),\mathrm{TPR}(t)\bigr) = (1,1).}
\]
Extreme cases
ROC curve for a perfect classifier
Theorem 1 (ROC curve for a perfect classifier) Suppose there exists a threshold \(t^\star \in \mathbb{R}\) such that \[
\mathbb{P}(X_0 \ge t^\star) = 0, \quad \mathbb{P}(X_1 \ge t^\star) = 1.
\] Then, \[
\boxed{\mathrm{FPR}(t^\star) = 0, \quad \mathrm{TPR}(t^\star) = 1,}
\] and the ROC curve passes through the upper-left corner \((0,1)\), in addition to the endpoints \((0,0)\) and \((1,1)\).
▶Proof
By definition, \(\mathrm{FPR}(t^\star) = \mathbb{P}(X_0 \ge t^\star) = 0\) and \(\mathrm{TPR}(t^\star) = \mathbb{P}(X_1 \ge t^\star) = 1\). Thus the threshold \(t^\star\) yields the operating point \((0,1)\), representing perfect discrimination.
ROC curve for an always-wrong classifier
Corollary 1 (ROC curve for an always-wrong classifier) Suppose there exists a threshold \(t^\star \in \mathbb{R}\) such that \[
\mathbb{P}(X_0 \ge t^\star) = 1, \quad \mathbb{P}(X_1 \ge t^\star) = 0.
\] Then, \[
\boxed{\mathrm{FPR}(t^\star) = 1, \quad \mathrm{TPR}(t^\star) = 0,}
\] and the ROC curve passes through the lower-right corner \((1,0)\), in addition to the endpoints \((0,0)\) and \((1,1)\).
▶Proof
By definition, \(\mathrm{FPR}(t^\star) = \mathbb{P}(X_0 \ge t^\star) = 1\) and \(\mathrm{TPR}(t^\star) = \mathbb{P}(X_1 \ge t^\star) = 0\). Thus the threshold \(t^\star\) yields the operating point \((1,0)\), representing perfectly inverted discrimination.
ROC curve under random guessing
Theorem 2 (ROC curve under random guessing) Suppose \(X_0 \stackrel{d}{=} X_1\). Then, for every \(t \in \mathbb{R}\), \[
\boxed{\mathrm{FPR}(t) = \mathrm{TPR}(t).}
\] Consequently, the ROC curve lies on the diagonal line \(u=v\) of the unit square.
▶Proof
Since \(X_0 \stackrel{d}{=} X_1\), \(\mathbb{P}(X_0 \ge t) = \mathbb{P}(X_1 \ge t)\) for every \(t \in \mathbb{R}\), so \(\mathrm{FPR}(t) = \mathrm{TPR}(t)\) for all \(t\), and the ROC curve coincides with the diagonal.
Monotonicity of ROC curves
A fundamental property of ROC curves, one that distinguishes them from Precision-Recall curves, is their monotonicity.
Theorem 3 (Monotonicity of ROC curves) Both \(\mathrm{TPR}(\cdot)\) and \(\mathrm{FPR}(\cdot)\) are non-increasing in the threshold: for any \(t_1, t_2 \in \mathbb{R}\) with \(t_1 < t_2\), \[
\begin{align*}
\mathrm{TPR}(t_1) & \ge \mathrm{TPR}(t_2),\\
\mathrm{FPR}(t_1) & \ge \mathrm{FPR}(t_2).
\end{align*}
\]
▶Proof
Recall \(\mathrm{TPR}(t) = \mathbb{P}(X_1 \ge t)\) and \(\mathrm{FPR}(t) = \mathbb{P}(X_0 \ge t)\). Fix \(t_1<t_2\): since \(X\ge t_2\) implies \(X\ge t_1\), the event \(\{X\ge t_2\}\) is contained in \(\{X\ge t_1\}\) for both \(X_0\) and \(X_1\). By monotonicity of probability measures, \[
\mathbb{P}(X_1 \ge t_2) \le \mathbb{P}(X_1 \ge t_1) \quad\text{and}\quad \mathbb{P}(X_0 \ge t_2) \le \mathbb{P}(X_0 \ge t_1),
\] i.e. \(\mathrm{TPR}(t_1) \ge \mathrm{TPR}(t_2)\) and \(\mathrm{FPR}(t_1) \ge \mathrm{FPR}(t_2)\).
Invariance of ROC curves under increasing transformations
A strictly increasing bijection does not introduce new ROC geometry: the ROC curve of \(g(X)\) is identical to that of \(X\), with only the threshold parametrization transformed via \(t\mapsto g^{-1}(t)\).
Theorem 4 (Invariance of ROC curves under increasing transformations) Let \(X\) be a real-valued random variable, with \(X\mid Y=0\) and \(X\mid Y=1\) atomless, and let \(g:\mathbb{R}\to\mathbb{R}\) be a strictly increasing bijection. Writing
\[
\begin{align*}
\mathrm{FPR}_X(t)=\mathbb{P}(X\ge t\mid Y=0),\\ \mathrm{TPR}_X(t)=\mathbb{P}(X\ge t\mid Y=1),
\end{align*}
\] the operating characteristics of \(g(X)\) are \[
\boxed{
\begin{aligned}
\mathrm{FPR}_{g(X)}(t) &= \mathrm{FPR}_X(g^{-1}(t)), \\
\mathrm{TPR}_{g(X)}(t) &= \mathrm{TPR}_X(g^{-1}(t)).
\end{aligned}
}
\]
▶Proof
Since \(g\) is a strictly increasing bijection, so is \(g^{-1}\), and the events \(\{g(X)\ge t\}\) and \(\{X\ge g^{-1}(t)\}\) coincide: \(g(X)\ge t \iff X\ge g^{-1}(t)\). Hence, for \(y\in\{0,1\}\), \[
\mathbb{P}(g(X)\ge t\mid Y=y) = \mathbb{P}(X\ge g^{-1}(t)\mid Y=y),
\] which, specializing to \(y=0\) and \(y=1\), is exactly the boxed identity. As \(t\) ranges over \(\mathbb{R}\), so does \(g^{-1}(t)\), and \(t\mapsto g^{-1}(t)\) is an order-preserving reparametrization of thresholds — so the ROC curve of \(g(X)\) is identical to that of \(X\), only traced by a relabeled threshold.
Symmetries of ROC curves
The ROC curve exhibits three nontrivial symmetries, generated by reversing the decision rule and/or swapping the class labels. Each acts as an exact affine reflection of the \((\mathrm{FPR},\mathrm{TPR})\) square, mostly because the ROC curve is completely unaffected by class prevalence.
Each symmetry is described by its action on the operating point \((\mathrm{FPR}(t),\mathrm{TPR}(t))\) of the upper-tail rule \(\hat{Y}_t=\mathbb{1}\{X\ge t\}\). No assumption on the laws of \(X\mid Y\) is needed here, since \(\{X<t\}\) is exactly the complement of \(\{X\ge t\}\).
Threshold reversal
Keeping the class labels unchanged and reversing the rule to \(\hat{Y}_t^\star=\mathbb{1}\{X<t\}\) sends the operating point as \[
\boxed{
\binom{\mathrm{FPR}(t)}{\mathrm{TPR}(t)} \longmapsto \binom{1-\mathrm{FPR}(t)}{1-\mathrm{TPR}(t)}.
}
\]
Theorem 5 (Central symmetry) The ROC curve under threshold reversal (class labels fixed) is the central reflection of the original ROC curve about \((0.5,0.5)\).
▶Proof
Under the reversed rule \(\hat{Y}_t^\star=\mathbb{1}\{X<t\}\), \[
\mathbb{P}(X<t\mid Y=0) = 1-\mathrm{FPR}(t), \quad \mathbb{P}(X<t\mid Y=1) = 1-\mathrm{TPR}(t).
\] Thus \((\mathrm{FPR}(t),\mathrm{TPR}(t))\mapsto(1-\mathrm{FPR}(t),\,1-\mathrm{TPR}(t))\), a central reflection about \((0.5,0.5)\).
Class label swap
Keeping the rule unchanged but swapping the labels, \(Y^\star=1-Y\), \(\hat{Y}_t^\star=\mathbb{1}\{X\ge t\}\), sends the operating point as \[
\boxed{
\binom{\mathrm{FPR}(t)}{\mathrm{TPR}(t)} \longmapsto \binom{\mathrm{TPR}(t)}{\mathrm{FPR}(t)}.
}
\]
Theorem 6 (Main-diagonal symmetry) Under class-label swapping with the rule unchanged, the ROC curve is the reflection of the original curve about the main diagonal \(y=x\).
▶Proof
Since \(Y^\star=1-Y\), \(\{Y^\star=0\}=\{Y=1\}\) and \(\{Y^\star=1\}=\{Y=0\}\). Under the unchanged rule, \[
\mathbb{P}(X\ge t\mid Y^\star=0) = \mathrm{TPR}(t), \quad \mathbb{P}(X\ge t\mid Y^\star=1) = \mathrm{FPR}(t).
\] Thus \((\mathrm{FPR}(t),\mathrm{TPR}(t))\mapsto(\mathrm{TPR}(t),\,\mathrm{FPR}(t))\), a reflection about \(y=x\).
Class label swap with threshold reversal
Swapping the labels and reversing the rule, \(Y^\star=1-Y\), \(\hat{Y}_t^\star=\mathbb{1}\{X<t\}\), sends the operating point as \[
\boxed{
\begin{pmatrix}\mathrm{FPR}(t)\\ \mathrm{TPR}(t)\end{pmatrix}
\longmapsto
\begin{pmatrix}1-\mathrm{TPR}(t)\\ 1-\mathrm{FPR}(t)\end{pmatrix}.
}
\]
Theorem 7 (Anti-diagonal symmetry) Under simultaneous class-label swapping and threshold reversal, the ROC curve is the reflection of the original curve about the anti-diagonal \(y=1-x\).
▶Proof
As above, \(\{Y^\star=0\}=\{Y=1\}\) and \(\{Y^\star=1\}=\{Y=0\}\). Under the reversed rule, \(\{X<t\}=\{X\ge t\}^c\) gives \[
\mathbb{P}(X<t\mid Y^\star=0) = 1-\mathrm{TPR}(t), \quad \mathbb{P}(X<t\mid Y^\star=1) = 1-\mathrm{FPR}(t).
\] Thus \((\mathrm{FPR}(t),\mathrm{TPR}(t))\mapsto(1-\mathrm{TPR}(t),\,1-\mathrm{FPR}(t))\), a reflection about \(y=1-x\).
Summary and visualization
Table 1: Transformations of the ROC operating point. Reversal: \(X<t \leftrightarrow X\ge t\); label swap: \(Y=0\leftrightarrow Y=1\).
Transformation
FPR\('\) / TPR\('\)
Geometry
Threshold reversal
\(1-\mathrm{FPR}\) / \(1-\mathrm{TPR}\)
Central symmetry, center \((0.5,0.5)\)
Label swap
\(\mathrm{TPR}\) / \(\mathrm{FPR}\)
Main diagonal \(y=x\)
Reversal + swap
\(1-\mathrm{TPR}\) / \(1-\mathrm{FPR}\)
Anti-diagonal \(y=1-x\)
These three transformations, together with the identity, preserve the curve’s overall shape. As noted above, none of them depends on the class prevalence \(\pi_1\). You can verify this directly with the interactive below
Interactive ROC curve exploration. The visualization below illustrates these properties for a Gaussian model \(X\mid Y=0\sim\mathcal N(0,1)\), \(X\mid Y=1\sim\mathcal N(1,2^2)\). Use the Threshold slider to move along the ROC curve and watch the operating point change; the Prevalence (\(\pi_1\)) slider confirms that the ROC curve itself is unaffected by class imbalance, however that slider is moved. The plot also shows the reversed-rule and class-inverted curves described above.
Figure 1: ROC curve under symmetry transformations
ROC curves for common distributions
Below we summarize the ROC curves for classic location-scale distributions, along with the exponential case. Throughout, we assume that \(Z_0, Z_1\) are independent and identically distributed with a common continuous CDF \(F\). We define the location-scale variables as:
\[
\begin{align*}
X_0 & = \sigma_0 Z_0 + \mu_0,\\
X_1 & = \sigma_1 Z_1 + \mu_1,
\end{align*}
\] where \(\sigma_0 > 0\) and \(\sigma_1 > 0\) are the scale parameters, and \(\mu_0, \mu_1 \in \mathbb{R}\) are the location parameters. We use the following notation:
Where the table comes from. All the rows follow from a single identity (Theorem 18): for \(X_y=\sigma_yZ_y+\mu_y\) with \(Z_0,Z_1\) i.i.d. with CDF \(F\), \[
\mathrm{TPR}\!=\!1\!-\!F\bigl(\rho\,F^{-1}(1\!-\!\mathrm{FPR})\!-\!\Delta\bigr),
\] which simplifies to \(\mathrm{TPR}=F\bigl(\Delta+\rho\,F^{-1}(\mathrm{FPR})\bigr)\) when \(F\) is symmetric (Corollary 3). The statements, the proofs, and the row-by-row computations (Gaussian, Cauchy, Logistic, Uniform, Gumbel, Exponential) are collected in Appendix A.
Visualization
The widget below lets you explore Theorem 18 interactively for the distribution described in Table 2. Use the sliders on the left to set \(\mu_0,\sigma_0\) (healthy class) and \(\mu_1,\sigma_1\) (sick class), and drag the Threshold slider to move \(t\).
The middle panel shows the two population densities plotted vertically: \(X_1\) (sick) on the left in blue, \(X_0\) (healthy) on the right in red. The horizontal black line marks the current threshold \(t\); the darker-shaded region of each density is \(\{X<t\}\) (the portion of that class predicted negative under the rule \(\hat{Y}_t=\mathbb{1}\{X\ge t\}\)) while the lighter region above the line is predicted positive.
The right panel shows the theoretical ROC curve for the current \((\mu_0,\sigma_0,\mu_1,\sigma_1)\), with the dashed diagonal as the random-guessing reference (Theorem 2) and a marker at the operating point \((\mathrm{FPR}(t),\mathrm{TPR}(t))\) corresponding to the threshold you’ve selected, so you can see exactly which point on the curve a given threshold produces, and watch the curve itself deform as you change the means, standard deviations, or their ratio \(\rho=\sigma_0/\sigma_1\).
The title above the plot reports the current \(\Delta=(\mu_1-\mu_0)/\sigma_1\), \(\rho = \sigma_0/\sigma_1\).
Figure 2: ROC curve for classic distributions
Note: Visually, the TPR is the ratio of the light blue area over the full blue area on the left, while the FPR is the ratio of the light red area over the full red area (right).
Interpretation
\(\Delta=(\mu_1-\mu_0)/\sigma_1\) measures the (scale-standardized) separation between \(X_0\) and \(X_1\); \(\rho=\sigma_0/\sigma_1\) compares their spread. Together they set the difficulty and shape of the ROC curve.
\(\rho=1\): the usual equal-scale model, \(\mathrm{TPR}=F(\Delta+F^{-1}(\mathrm{FPR}))\) for symmetric \(F\), e.g., the binormal case \(\mathrm{TPR}=\Phi(\Delta+\Phi^{-1}(\mathrm{FPR}))\).
\(\Delta=0,\ \rho=1\):\(X_0\stackrel{d}{=}X_1\), and the ROC curve is the diagonal (random guessing). Note \(\Delta=0\) alone is not enough when \(\rho\ne1\): equal location with unequal scale still separates the classes, via spread rather than location.
\(\rho\ne1\): the curve need not be symmetric about the anti-diagonal or concave everywhere, if \(\rho<1\) (sick population more dispersed), the curve can dip toward the diagonal near one corner before rising, a non-concave S-shape sometimes seen in real assay data.
\(\Delta\to\infty\) (fixed \(\rho\)): the classes separate perfectly, and the curve approaches the ideal curve through \((0,1)\).
Now we illustrate the empirical counterpart, when one samples elements from the distributions but without access to the true densities.
Figure 3: Empirical counterpart of ROC curve for classic distributions.
Note: the jittering on the x-axis for the scatter plots is not meaningful but only here for visualization.
Area Under the ROC Curve (Cis ranked higher than a randomly chosen negative one, ties counting for one half.
Often the AUC is used to measure overall performance: the larger the AUC, the better the assay discriminates between the two classes.
Remind that \(X_0\) and \(X_1\) are independent, distributed as the conditional laws of \(X\) given \(Y=0\) and \(Y=1\), respectively: \[
\begin{align*}
X_0 & \sim \mathcal{L}(X\mid Y=0)\\
X_1 & \sim \mathcal{L}(X\mid Y=1).
\end{align*}
\] As conditional laws of a real random variable \(X\), both \(X_0\) and \(X_1\) are almost surely finite. Below we use the notation \(\overline{\mathbb{R}}= [-\infty, + \infty]\).
Define the randomized decision rule \[
\phi_s(x) =
\begin{cases}
1, & x > t(s), \\
\lambda(s), & x = t(s), \\
0, & x < t(s).
\end{cases}
\]
Then \(\lambda(s)\in[0,1]\) is well defined for every \(s\in[0,1]\), and \[
\mathrm{FPR}(\phi_s) \triangleq \mathbb{E}[\phi_s(X_0)] = s,
\] so the (randomized) ROC curve \(\mathrm{ROC}(s)\triangleq\mathrm{TPR}(\phi_s)=\mathbb{E}[\phi_s(X_1)]\), \(s\in[0,1]\), is well defined for any laws of \(X_0\) and \(X_1\).
Moreover, at every atom \(c\) of \(X_0\) (i.e., \(\mathbb{P}(X_0=c)>0\)), \(\mathrm{ROC}\) is affine on the interval \(\bigl(\mathbb{P}(X_0>c),\ \mathbb{P}(X_0\ge c)\bigr]\), of length \(\mathbb{P}(X_0=c)\), with slope \(\mathbb{P}(X_1=c)/\mathbb{P}(X_0=c)\).
Throughout, write \(S_0(t)\triangleq\mathbb{P}(X_0\ge t)\) for the survival function of \(X_0\), and \(S_0(x+)\triangleq\lim_{u\downarrow x}S_0(u)\) for its right limit at any \(x\in\mathbb{R}\). As a survival function, \(S_0\) is non-increasing, and it is left-continuous, \(S_0(x)=\lim_{u\uparrow x}S_0(u)\), since \(\{X_0\ge u_n\}\downarrow\{X_0\ge x\}\) along any \(u_n\uparrow x\); likewise \(\{X_0\ge u_n\}\uparrow\{X_0>x\}\) along any \(u_n\downarrow x\) gives \(S_0(x+)=\mathbb{P}(X_0>x)\), so \[
\mathbb{P}(X_0=x)\triangleq S_0(x)-S_0(x+)=\mathbb{P}(X_0=x)
\] is exactly the atomic mass of \(X_0\) at \(x\). Since \(X_0\) is a.s. finite, continuity of measure applied to the nested events \(\{X_0\ge t\}\) as \(t\to\mp\infty\) also gives \[
\lim_{t\to-\infty}S_0(t)=1, \quad \lim_{t\to\infty}S_0(t)=0.
\]
Step 1 (existence of \(t(s)\)). Call \(L\subseteq\mathbb{R}\) a lower set if \(t\in L,\,u\le t\Rightarrow u\in L\). Fix \(s\in(0,1)\) and set \(A_s\triangleq\{t:S_0(t)\ge s\}\). Since \(S_0\) is non-increasing, \(A_s\) is a lower set; since \(S_0\to1>s\) as \(t\to-\infty\) and \(S_0\to0<s\) as \(t\to\infty\), \(A_s\) is nonempty and bounded above, so \[
t(s)\triangleq\sup A_s\ \in\ \mathbb{R}
\] is well defined. In fact \(A_s=(-\infty,t(s)]\): any \(t<t(s)\) is not an upper bound of \(A_s\), hence lies in \(A_s\) by the lower-set property; and \(t(s)\in A_s\) by left-continuity of \(S_0\) along \(t_n\uparrow t(s)\), \(t_n\in A_s\).
For the endpoints \(s\in\{0,1\}\), the same set \(A_s=\{t:S_0(t)\ge s\}\) is used, with the convention \(\sup\emptyset=-\infty\): since \(S_0\le1\) everywhere, \(A_0=\mathbb{R}\) and \(t(0)=+\infty\); and \(A_1=\{t:S_0(t)=1\}\), which is a (possibly empty) lower set, giving \(t(1)\in\mathbb{R}\cup\{-\infty\}\) with \(A_1=(-\infty,t(1)]\) by the same argument (or \(A_1=\emptyset\), \(t(1)=-\infty\), if \(S_0(t)<1\) for every \(t\)).
Step 2 (level sets, and the defining inequality). For \(x\in\mathbb{R}\), \[
t(s)=x \quad\Longleftrightarrow\quad S_0(x+) < s \le S_0(x). \tag{$\dagger$}
\]
Proof. By Step 1, \(t(s)=x\) if and only if \(x\in A_s\) and no \(y>x\) lies in \(A_s\), i.e., \[
S_0(x)\ge s \quad\text{and}\quad S_0(y)<s\ \ \forall y>x.
\] Since \(S_0\) is non-increasing, \(\sup_{y>x}S_0(y)=S_0(x+)\), so the second condition is equivalent to \(s>S_0(x+)\); this proves \((\dagger)\).
Taking \(x=t(s)\) in \((\dagger)\) yields both the defining inequality and the atom size at the threshold: \[
\begin{cases}
S_0(t(s)+) \ \le\ s\ \le\ S_0(t(s)), \\
S_0(t(s))-S_0(t(s)+) = \mathbb{P}(X_0=t(s)). \tag{$\ast$}
\end{cases}
\]
Step 3 (\(\phi_s\) attains \(\mathrm{FPR}(\phi_s)=s\) exactly). By \((\ast)\), \[
0\ \le\ s-S_0(t(s)+)\ \le\ \mathbb{P}(X_0=t(s)),
\] so whenever \(\mathbb{P}(X_0=t(s))>0\), \(\lambda(s)=\dfrac{s-S_0(t(s)+)}{P(X_0=t(s))}\in[0,1]\) . If instead \(\mathbb{P}(X_0=t(s))=0\), the two bounds coincide and force \(s=S_0(t(s)+)\); the definition sets \(\lambda(s)\triangleq 0\) in this case, and its value has no impact since it multiplies the null event \(\{X_0=t(s)\}\). In either case, \[
\mathrm{FPR}(\phi_s) = \mathbb{E}[\phi_s(X_0)] = S_0(t(s)+) + \lambda(s)\,\mathbb{P}(X_0=t(s)) = s.
\]
Step 4 (affine property of ROC). Fix \(c\) with \(\mathbb{P}(X_0=c)>0\), and let \(s\in(S_0(c+),S_0(c)]\). By \((\dagger)\), \(t(s)=c\) throughout this interval, so substituting \(t(s)=c\) into the definition of \(\lambda\), legitimate since \(\mathbb{P}(X_0=t(s))=\mathbb{P}(X_0=c)>0\) there, gives \[
\lambda(s) = \frac{s-S_0(c+)}{P(X_0=c)},
\] a linear, increasing function of \(s\), running from \(0\) at \(s=S_0(c+)\) to \(1\) at \(s=S_0(c)\). Hence \(\mathrm{ROC}(s) = \mathbb{E}[\phi_s(X_1)] = \mathbb{P}(X_1>c) + \lambda(s)\,\mathbb{P}(X_1=c)\) is affine in \(s\) on \((S_0(c+),S_0(c)]\), an interval of length \(\mathbb{P}(X_0=c)\), with slope \(\mathbb{P}(X_1=c)/\mathbb{P}(X_0=c)\).
Step 5 (AUC). Since \(t(\cdot)\) is non-increasing (its sublevel sets \(A_s\) are nested in \(s\), by Step 1) and \(\lambda(\cdot)\) is Borel where defined, \((s,x)\mapsto\phi_s(x)\) is jointly measurable, \([0,1]\)-valued, so Tonelli’s theorem gives \[
\mathrm{AUC} = \int_0^1 \mathbb{E}[\phi_s(X_1)]\,ds = \mathbb{E}\!\left[\int_0^1 \phi_s(X_1)\,ds\right].
\]
Fix \(x\in\mathbb{R}\). By \((\dagger)\), up to the single (Lebesgue-null) point \(S_0(x+)\), the interval \([0,1]\) splits into three sets, according to the position of \(t(s)\) relative to \(x\): \[
\begin{align*}
\{s:t(s)>x\} &= [0,S_0(x+)], & \operatorname{Leb}\bigl(\{s:t(s)>x\}\bigr) &= \mathbb{P}(X_0>x), \\
\{s:t(s)=x\} &= (S_0(x+),S_0(x)], & \operatorname{Leb}\bigl(\{s:t(s)=x\}\bigr) &= \mathbb{P}(X_0=x), \\
\{s:t(s)<x\} &= (S_0(x),1], & \operatorname{Leb}\bigl(\{s:t(s)<x\}\bigr) &= \mathbb{P}(X_0<x).
\end{align*}
\]
On each of these three sets, \(\phi_s(x)=\mathbb{1}\{x>t(s)\}+\lambda(s)\mathbb{1}\{x=t(s)\}\) takes a simple form: \[
\phi_s(x) =
\begin{cases}
0, & t(s)>x,\\
\lambda(s), & t(s)=x,\\
1, & t(s)<x.
\end{cases}
\] Hence \(\int_0^1\phi_s(x)\,ds\) splits into three integrals, one over each set: \[
\int_0^1\phi_s(x)\,ds
= \underbrace{\int_{\{t(s)>x\}} 0\,ds}_{=\,0}
\ +\ \underbrace{\int_{\{t(s)=x\}} \lambda(s)\,ds}_{=:I(x)}
\ +\ \underbrace{\int_{\{t(s)<x\}} 1\,ds}_{=\,\mathbb{P}(X_0<x)}.
\]
It remains to compute \(I(x)=\int_{(S_0(x+),S_0(x)]} \lambda(s)\,ds\), by splitting on whether \(x\) is an atom of \(X_0\):
If \(\mathbb{P}(X_0=x)=0\): the domain of integration is Lebesgue-null, so \(I(x)=0=\tfrac12\,\mathbb{P}(X_0=x)\).
If \(\mathbb{P}(X_0=x)>0\): by Step 4 (with \(c=x\)), \(\lambda(s)\) runs linearly from \(0\) to \(1\) as \(s\) ranges over \((S_0(x+),S_0(x)]\). The average of a function running linearly between \(0\) and \(1\) over an interval is \(\tfrac12\) (the midpoint of its range), so \(I(x)=\mathbb{P}(X_0=x)\cdot\tfrac12\).
Either way, \(I(x)=\tfrac12\,\mathbb{P}(X_0=x)\), and therefore \[
\int_0^1 \phi_s(x)\,ds = \mathbb{P}(X_0<x) + \tfrac12\,\mathbb{P}(X_0=x) = \mathbb{P}(X_0<x) + \tfrac12\,\mathbb{P}(X_0=x).
\]
Substituting \(x=X_1\), taking expectations, and using the tower property, \[
\mathrm{AUC} = \mathbb{P}(X_0<X_1) + \tfrac12\mathbb{P}(X_0=X_1) = \mathbb{P}(X_1>X_0) + \tfrac12\mathbb{P}(X_1=X_0).
\]
For continuous random variables (with no point masses), the AUC reduces to the probability that \(X_1 > X_0\).
Corollary 2 (AUC for continuous distributions) Let \(X_0 \sim \mathcal{L}(X \mid Y=0)\) and \(X_1 \sim \mathcal{L}(X \mid Y=1)\) be independent continuous random variables (no point masses).
Replacing \(X_0,X_1\) by two independent samples and \(\mathbb{P}(X_1>X_0)\) by the corresponding sample proportion gives exactly the classical Mann-Whitney \(U\) statistic from nonparametric statistics, providing the empirical, plug-in counterpart to Corollary 2.
Theorem 9 (Empirical AUC and Mann-Whitney U Statistic) Let \(X_{1,1}, \dots, X_{1,n_1}\) be an i.i.d. sample from \(X_1 \sim \mathcal{L}(X \mid Y=1)\) and \(X_{0,1}, \dots, X_{0,n_0}\) be an i.i.d. sample from \(X_0 \sim \mathcal{L}(X \mid Y=0)\), with both samples independent of each other. Assume \(X_1\) and \(X_0\) are continuous random variables (no point masses).
Let \(R_1\) be the sum of the ranks of the \(X_{1,i}\) values in the combined sample of size \(n_1 + n_0\) (ranks assigned in ascending order, so the smallest value has rank \(1\)). The Mann-Whitney U statistic for the positive group is defined as: \[
\begin{align*}
U_1 & = R_1 - \frac{n_1 (n_1 + 1)}{2}\\
& = \sum_{i=1}^{n_1}\sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}).
\end{align*}
\]
Then, the empirical AUC of the ROC curve is equivalent to the normalized Mann-Whitney U statistic: \[
\boxed{
\widehat{\mathrm{AUC}} = \frac{1}{n_1 n_0}\sum_{i=1}^{n_1}\sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}).
}
\]
▶Proof
Rank Sum Interpretation: The term \(R_1\) in \(U_1\) is the sum of the ranks of the \(X_{1,i}\) values in the combined sample of size \(n_1 + n_0\). For continuous distributions, all \(X_{1,i}\) and \(X_{0,j}\) are distinct almost surely, so the rank of each \(X_{1,i}\) counts: the \(X_{0,j}\) values it exceeds, the \(X_{1,i'}\) values (\(i' \neq i\)) it exceeds, and \(1\) for its own position. Introducing \[
\begin{align*}
S_i & \triangleq \sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}),\\
C_i & \triangleq \sum_{i' \neq i} \mathbb{1}(X_{1,i} > X_{1,i'}),
\end{align*}
\] this reads \[
R_1 = \sum_{i=1}^{n_1} \left( S_i + C_i + 1 \right).
\]
Simplification of \(R_1\): For each unordered pair \(\{i,i'\}\subseteq\{1,\dots,n_1\}\), exactly one of the two indices has the larger value, so \(\sum_i C_i\) counts each of the \(\binom{n_1}{2}\) pairs exactly once: \[
\sum_{i=1}^{n_1} C_i = \binom{n_1}{2} = \frac{n_1(n_1-1)}{2}.
\] Writing \[
S \triangleq \sum_{i=1}^{n_1}\sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j})
\] for the number of cross-sample pairs where the positive-group value is larger, this gives \[
R_1 = S + \frac{n_1(n_1-1)}{2} + n_1 = S + \frac{n_1(n_1+1)}{2}.
\]
Link to \(U_1\): Substituting this expression for \(R_1\) into the definition of \(U_1\), \[
U_1 = \left(S + \frac{n_1(n_1+1)}{2}\right) - \frac{n_1(n_1+1)}{2} = S,
\] so \(U_1\) is exactly the count of cross-sample pairs with \(X_{1,i}>X_{0,j}\): \[
U_1 = \sum_{i=1}^{n_1} \sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}).
\]
Empirical AUC: Recall that \(\widehat{\mathrm{AUC}}\) is defined as the fraction of positive/negative pairs correctly ordered by the score: \[
\widehat{\mathrm{AUC}} = \frac{1}{n_1 n_0} \sum_{i=1}^{n_1} \sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}) = \frac{S}{n_1n_0}.
\] By Step 3, \(S=U_1\), so \[
\widehat{\mathrm{AUC}} = \frac{U_1}{n_1 n_0}.
\]
We now determine how the three symmetries of the previous section act on the AUC.
Theorem 10 (AUC under threshold reversal and class inversion) Assume \(X\mid Y=0\) and \(X\mid Y=1\) are atomless and almost surely finite. Write \(\mathrm{AUC}^\star\) for the AUC of the transformed problem in each case below. Then:
under threshold reversal alone, \(\boxed{\mathrm{AUC}^\star = 1-\mathrm{AUC}}\);
under class label swap alone, \(\boxed{\mathrm{AUC}^\star = 1-\mathrm{AUC}}\);
under both together (the anti-diagonal transform), \(\boxed{\mathrm{AUC}^\star = \mathrm{AUC}}\), leaving the AUC unchanged.
▶Proof
Write \(u\triangleq\mathrm{FPR}(t)\), \(f(u)\triangleq\mathrm{TPR}(t)\), so \(\mathrm{AUC}=\int_0^1 f(u)\,du\), with \(f\) non-decreasing (Theorem 3), \(f(0)=0\), \(f(1)=1\) (Lemma 1). Let \(R\triangleq\{(x,y):0\le x\le1,\,0\le y\le f(x)\}\), of area \(\mathrm{AUC}\).
Threshold reversal. The transformed curve is \(g(x')\triangleq 1-f(1-x')\). Reflecting \(R\) through \((0.5,0.5)\) sends \((x,y)\mapsto(1-x,1-y)\), and \(0\le y\le f(x)\) becomes \(1-f(1-x')\le y'\le1\), the region above\(g\), of area \(1-\mathrm{AUC}^\star\). Since central reflection preserves area, \(1-\mathrm{AUC}^\star=\mathrm{AUC}\), i.e., \(\mathrm{AUC}^\star=1-\mathrm{AUC}\).
Class label swap. The transformed curve is \(g\triangleq f^{-1}\) (generalized inverse). Reflecting \(R\) across \(y=x\) sends \((x,y)\mapsto(y,x)\), and \(0\le x\le f(y)\) becomes, by monotonicity of \(f\), \(y\ge f^{-1}(x)=g(x)\), the same region above \(g\), of area \(1-\mathrm{AUC}^\star\). Reflection across a line preserves area, so once more \(\mathrm{AUC}^\star=1-\mathrm{AUC}\).
Both together. The two transforms commute at the level of the operating point, so applying one after the other composes the two identities just derived: \(\mathrm{AUC}\mapsto1-\mathrm{AUC}\mapsto1-(1-\mathrm{AUC})=\mathrm{AUC}\).
Interpretation. Only the composite (anti-diagonal) transform preserves AUC exactly; the two individual transforms each send \(A\mapsto1-A\). This is intuitive: reversing the rule alone, or swapping the labels alone, each turns a good classifier into a bad one for the new problem (a classifier that correctly separates \(X_0\) from \(X_1\) under the original labeling is simply mislabeling everything once only one of the two conventions is flipped). On the other hand, flipping both conventions together restores consistency, since relabeling “positive” to mean “low score” and simultaneously testing for “low score” recovers a description of the exact same underlying discrimination ability. The fixed point of \(A\mapsto1-A\) at \(A=\tfrac12\) (random guessing) is the one case where all three transforms leave AUC unchanged, since \(1-\tfrac12=\tfrac12\).
You can verify this directly with the interactive above, see Figure 1: the “ROC curve” and “Class swap + threshold rev.” curves in the plot share the same AUC, while “Threshold reversal” and “Class label swap” alone each trace out the complementary area.
The precision–recall (PR) curve
The precision–recall (PR) curve has a different origin from the ROC curve: it comes from information retrieval, not from signal detection. In the 1950s, the first automated literature-searching systems raised a practical question: how should one measure the quality of a search? Kent et al. (1955) proposed to count the share of the relevant documents that are retrieved (what we now call recall) and the share of the retrieved documents that are relevant (a first version of precision, then called “pertinency”). Medicine had its own names for the same quantities: recall is the sensitivity, and precision is the positive predictive value.
Definition 2 (Precision–recall (PR) curve) For a threshold \(t\in\mathbb{R}\), the precision–recall (PR) operating point is the pair of recall and precision,
where \(\hat{Y}_t=\mathbb{1}\{X\geq t\}\) denotes the threshold-based classifier. The PR curve is the parametric curve traced by this operating point as \(t\) varies over \(\mathbb{R}\):
Unlike recall (=TPR) and the FPR, which are conditional on the true class, precision depends on the prevalence \(\pi_1\) of the positive class in the population being evaluated. Consequently, the PR curve can change when prevalence changes, even if the class-conditional score distributions remain the same.
The empirical recall \(\widehat{\mathrm{Recall}}(t)=\mathrm{TP}(t)/n_1\) and precision \(\widehat{\mathrm{Precision}}(t)=\mathrm{TP}(t)/\hat{\mathrm{P}}_t\) are the plug-in estimators of the naming section (\(\widehat{\mathrm{Recall}}=\widehat{\mathrm{TPR}}\)). If no observation is predicted positive, then \(\hat{\mathrm{P}}_t=0\) and the empirical precision is undefined: implementations of PR curves adopt a convention for this endpoint, which should be considered when comparing empirical curves.
The following theorem makes explicit how prevalence enters the precision formula.
Theorem 11 (Precision in terms of TPR, FPR, and prevalence) Let \(\pi_1=\mathbb{P}(Y=1)\) and \(\pi_0=\mathbb{P}(Y=0)\). For every threshold \(t\) such that \(\mathbb{P}(X\ge t)>0\),
Substituting the expressions above yields the claimed formula.
For fixed class-conditional score distributions, \(\mathrm{TPR}(t)\) and \(\mathrm{FPR}(t)\) do not change with prevalence. The ROC curve therefore remains unchanged, whereas the PR curve generally changes through \(\pi_1\) and \(\pi_0\) in the theorem above. In particular, precision at a fixed threshold typically decreases as positive cases become less common.
In our COVID-19 example, the 976 pre-pandemic negative samples and 536 confirmed positive samples reflect the benchmark’s sampling design, not necessarily the prevalence of infection in a clinical population. Consequently, its empirical PR curve and average precision need not represent predictive performance in a target population with a different class composition. The initial dataset description gives the benchmark’s positive fraction; it should not be interpreted as clinical prevalence.
The Python code to compute this curve is provided below.
Code
def precision_recall(thresholds, y_true, score): precision = [] recall = [] P =sum(y_true)for thresh in thresholds: FP =0 TP =0for i inrange(len(score)):if score[i] >= thresh:if y_true[i] ==True: TP +=1else: FP +=1 recall.append(TP /float(P))# Convention: precision is set to 1 when nothing is predicted positive# (TP + FP = 0), as in scikit-learn at the recall = 0 endpoint. precision.append(TP /float(TP + FP) if (TP + FP) >0else1.0)return precision, recall
Properties
Endpoints
Throughout this section, assume \(\pi_1\in(0,1)\), \(\pi_0=1-\pi_1\), and that \(X_0,X_1\) are almost surely finite. Recall and precision are as defined above; precision is defined only when \(\mathbb{P}(X\ge t)>0\).
Lemma 2 (Universal endpoint of the PR curve) There exists \(t_0\in\mathbb{R}\) such that the denominator of \(\mathrm{Precision}(t)\) is positive for all \(t\le t_0\), and \[
\bigl(\mathrm{Recall}(t),\mathrm{Precision}(t)\bigr)\mathrel{\underset{t\to-\infty}{\longrightarrow}} (1,\pi_1).
\]
▶Proof
Since \(X_0\) and \(X_1\) are almost surely finite, the events \(\{X_y\ge t\}\) increase to \(\{X_y>-\infty\}\), which has probability \(1\), as \(t\downarrow-\infty\). By continuity of probability measures, \[
\lim_{t\to-\infty}\mathbb{P}(X_y\ge t)=1,\quad y\in\{0,1\}.
\] Hence \(\mathrm{Recall}(t)\to1\), and the denominator of \(\mathrm{Precision}(t)\) tends to \(\pi_1+\pi_0=1\). In particular it is positive for all \(t\) sufficiently negative, so \(\mathrm{Precision}(t)\) is defined there. Since the limit of the denominator is \(1\neq0\), the quotient rule for limits gives \[
\mathrm{Precision}(t)\longrightarrow\frac{\pi_1\cdot1}{\pi_1\cdot1+\pi_0\cdot1}=\pi_1.
\]
Remark (the other end, \(\mathrm{Recall}\to0\)). The limit as \(t\to+\infty\) is not universal: numerator and denominator of \(\mathrm{Precision}(t)\) both tend to \(0\), and \[
\mathrm{Precision}(t)=\Bigl(1+\frac{\pi_0}{\pi_1}\,\frac{\mathbb{P}(X_0\ge t)}{\mathbb{P}(X_1\ge t)}\Bigr)^{-1}
\] is governed by the tail ratio \(\mathbb{P}(X_0\ge t)/\mathbb{P}(X_1\ge t)\). For instance:
\(X_y\sim\mathcal N(\mu_y,\sigma^2)\) with \(\mu_1>\mu_0\): the ratio tends to \(0\), and the curve ends at \((0,1)\).
\(X_0\stackrel{d}{=}X_1\): the ratio is \(\equiv1\), and the curve ends at \((0,\pi_1)\).
\(X_y\sim\mathcal N(\mu_y,\sigma^2)\) with \(\mu_1<\mu_0\): the ratio tends to \(+\infty\), and the curve ends at \((0,0)\).
Extreme cases
Perfect classifier
Theorem 12 (PR curve for a perfect classifier) Suppose there exists a threshold \(t^\star \in \mathbb{R}\) such that \[
\mathbb{P}(X_0 \ge t^\star) = 0, \quad \mathbb{P}(X_1 \ge t^\star) = 1.
\] Then, \[
\boxed{\mathrm{Recall}(t^\star) = 1, \quad \mathrm{Precision}(t^\star) = 1,}
\] For every \(t\ge t^\star\) with \(\mathrm{Recall}(t)>0\), precision is \(1\). Thus the curve traces the horizontal segment at precision \(1\) from \((1,1)\) toward \((0,1)\); at a threshold with no predicted positives, population precision is undefined, although PR-curve implementations commonly use \(1\) for this endpoint. As \(t\to-\infty\), the curve also approaches the universal endpoint \((1,\pi_1)\).
▶Proof
The point at \(t^\star\). By definition, \[
\begin{align*}
\mathrm{Recall}(t^\star)
& = \mathbb{P}(X_1 \ge t^\star) = 1, \\
\mathrm{Precision}(t^\star)
& = \frac{\pi_1 \cdot 1}{\pi_1 \cdot 1 + \pi_0 \cdot 0} = 1.
\end{align*}
\]
The flat segment. By Theorem 3, \(\mathrm{FPR}(t)=\mathbb{P}(X_0\ge t)\) is non-increasing in \(t\). Since \(\mathrm{FPR}(t^\star)=0\) and \(\mathrm{FPR}\ge0\) always, monotonicity gives \(\mathrm{FPR}(t)\le\mathrm{FPR}(t^\star)=0\) for every \(t\ge t^\star\), so \[
\mathrm{FPR}(t) = 0 \quad\text{for all } t\ge t^\star.
\] Consequently, for any \(t\ge t^\star\) with \(\mathrm{TPR}(t)>0\), \[
\mathrm{Precision}(t) = \frac{\pi_1\,\mathrm{TPR}(t)}{\pi_1\,\mathrm{TPR}(t) + \pi_0\cdot0} = 1.
\] If \(\mathrm{TPR}(t)=0\) as well, then no positives are predicted and population precision is undefined. Thus the population PR curve has precision \(1\) on the nonempty predicted-positive part of this threshold range; empirical implementations may separately assign precision \(1\) at the no-predictions endpoint.
The endpoints. As \(t\) ranges over \([t^\star,\infty)\), recall decreases from \(1\) to \(0\), and precision remains \(1\) whenever predicted positives remain. The endpoint at recall \(0\) is represented as \((0,1)\) under the usual PR-plotting convention; population precision is undefined if no positives are predicted. As \(t\to-\infty\), \(\mathrm{TPR}(t)\to1\) and \(\mathrm{FPR}(t)\to1\) (Lemma 1), so precision tends to \(\pi_1\): the universal endpoint \((1,\pi_1)\) (see Theorem 11).
Always-wrong classifier
Theorem 13 pins down the PR curve only for \(t\ge t^\star\) (a single point, via monotonicity of \(\mathrm{TPR}\)). The same hypothesis, applied to \(\mathrm{FPR}\) instead, determines the remaining branch \(t\le t^\star\) as well.
Theorem 13 (PR curve for an always-wrong classifier) Suppose there exists \(t^\dagger \in \mathbb{R}\) such that \[
\mathbb{P}(X_0 \ge t^\dagger) = 1, \quad \mathbb{P}(X_1 \ge t^\dagger) = 0
\] Then the operating points satisfy the following:
for \(t\ge t^\dagger\): \(\mathrm{Recall}(t)\equiv0\); precision is \(0\) when \(\mathrm{FPR}(t)>0\) and is undefined when no positives are predicted;
for \(t\le t^\dagger\): \(\mathrm{FPR}(t)\equiv1\), and with \(r\triangleq \mathrm{Recall}(t)\), every attainable recall value satisfies \[
\boxed{\mathrm{Precision}(r) = \frac{\pi_1\,r}{\pi_1\,r + \pi_0}.}
\] If the positive-class score distribution is continuous, these points trace the full curve for \(r\in[0,1]\).
▶Proof
Branch \(t\ge t^\dagger\). By the assumption, \(\mathrm{TPR}(t^\dagger)=0\); monotonicity (Theorem 3) gives \(\mathrm{TPR}(t)=0\) for every \(t\ge t^\dagger\). Hence recall is \(0\) throughout this range. Precision is \(0\) when \(\mathrm{FPR}(t)>0\) and is undefined if \(\mathrm{FPR}(t)=0\), because then no positives are predicted. Under the common plotting convention that assigns precision \(1\) when no positives are predicted, the limiting endpoint is represented as \((0,1)\); the population definition itself leaves precision undefined there.
Branch \(t\le t^\dagger\). Since \(\mathrm{FPR}(t^\dagger)=1\) and \(\mathrm{FPR}\) is non-increasing, \(\mathrm{FPR}(t)\ge1\) for \(t\le t^\dagger\), forcing \(\mathrm{FPR}(t)\equiv1\) there. Substituting into Theorem 11, \[
\begin{align*}
\mathrm{Precision}(t)
& = \frac{\pi_1\,\mathrm{TPR}(t)}{\pi_1
\mathrm{TPR}(t)+\pi_0}\\
& = \frac{\pi_1\,r}{\pi_1\,r+\pi_0}, \quad r\triangleq \mathrm{Recall}(t).
\end{align*}
\] As \(t\) decreases from \(t^\dagger\) to \(-\infty\), \(r=\mathrm{TPR}(t)\) increases from \(0\) to \(1\) (Lemma 1), sweeping the boxed curve up to \((1,\pi_1)\).
Random guessing
Theorem 14 (PR curve under random guessing) Suppose \(X_0 \stackrel{d}{=} X_1\). Write \(p(t)\triangleq \mathbb{P}(X_0\ge t)=\mathbb{P}(X_1\ge t)\). Then \(\mathrm{Precision}(t)\) is defined if and only if \(p(t)>0\), and in that case \[
\boxed{\mathrm{Recall}(t)=p(t), \quad \mathrm{Precision}(t) = \pi_1.}
\] Consequently the PR curve \(\{(\mathrm{Recall}(t),\mathrm{Precision}(t)) : p(t)>0\}\) is contained in the horizontal line \(\mathrm{Precision}=\pi_1\).
▶Proof
By definition, \(\mathrm{Recall}(t)=\mathbb{P}(X_1\ge t)\) and \[
\begin{align*}
\mathrm{Precision}(t)
& =\mathbb{P}(Y=1\mid\hat{Y}_t=1)\\
& =\frac{\pi_1\,\mathbb{P}(X_1\ge t)}{\pi_1\,\mathbb{P}(X_1\ge t)+\pi_0\,\mathbb{P}(X_0\ge t)},
\end{align*}\] which is defined if and only if the denominator \(\mathbb{P}(\hat{Y}_t=1)\) is positive. Since \(X_0\stackrel d=X_1\), both tail probabilities equal \(p(t)\), so \(\mathrm{Recall}(t)=p(t)\) and the denominator equals \((\pi_1+\pi_0)\,p(t)=p(t)\). Hence \(\mathrm{Precision}(t)\) is defined if and only if \(p(t)>0\), and then \[
\mathrm{Precision}(t)=\frac{\pi_1\,p(t)}{p(t)}=\pi_1.
\]
Interpretation. The three cases show an asymmetry between the two ends of the PR curve.
\(\mathrm{Recall}=1\) is the universal endpoint. It is reached as \(t\to-\infty\), where the classifier labels everything positive. Precision is then simply the base rate, \(\mathrm{Precision}=\pi_1\), whatever the score distributions are (Lemma 1 + Theorem 11, no extra hypothesis). The perfect, random and always-wrong classifiers all pass through \((1,\pi_1)\).
\(\mathrm{Recall}=0\) is model-dependent. It is reached as \(t\to+\infty\), where the classifier labels almost nothing positive, and \(\mathrm{Precision}\) measures the purity of the extreme upper tail of the scores. That purity is not determined by \(\pi_1\) or by the ROC endpoints alone, but by the tail ratio \(\mathbb{P}(X_0\ge t)/\mathbb{P}(X_1\ge t)\). The limit can take any value in \([0,1]\): \(1\) for the perfect classifier (the top scores are all positives), \(\pi_1\) for random guessing (the top scores carry no information), and \(0\) for the always-wrong classifier (the top scores are all negatives).
This is why the earlier Gaussian and exponential derivations needed tail-rate arguments (Mills’ ratio; a power-law comparison) to determine the \(\mathrm{Recall}=0\) end. Unlike the \(\mathrm{Recall}=1\) endpoint, it is never given for free.
Symmetries of PR curves
Unlike in ROC space, none of the three transformations (threshold reversal, class label swap, and their composition) has a simple geometric description in PR space: there is no reflection or rotation of the original PR curve that produces the transformed curves. The map \((\mathrm{FPR},\mathrm{TPR})\mapsto(\mathrm{Recall},\mathrm{Precision})\) is nonlinear and depends on the prevalence \(\pi_1=\mathbb{P}(Y=1)\), and each transformation also changes the role of the positive class. Each transformed curve must therefore be recomputed from the definitions of precision and recall. Consequently, the transformed curves generally have a different shape from the original one, and, unlike the ROC curve, they all depend on \(\pi_1\). What remains simple is the endpoint at \(\mathrm{Recall}=1\): it is \((1,\pi_1)\) for the original curve and the threshold reversal, and \((1,1-\pi_1)\) for the two curves with swapped labels. You can verify this directly with the interactive below.
Interactive PR curve exploration. The visualization below illustrates these properties for the same Gaussian model \(X\mid Y=0\sim\mathcal N(0,1)\), \(X\mid Y=1\sim\mathcal N(1,2^2)\) as before. Use the Threshold (\(t\)) slider to move along the PR curve and watch the operating point change, together with the corresponding points on the transformed curves. The Prevalence (\(\pi_1\)) slider shows that, unlike the ROC curve, the PR curve and its transformations are reshaped by the class balance, and the endpoints at \(\mathrm{Recall}=1\) move with it. The plot also shows the threshold-reversed and class-swapped curves described above, as well as the extreme cases (perfect, random and always-wrong classifiers). Click an entry in the legend to show or hide the corresponding curve.
Figure 4: PR curve under symmetry transformations
PR curves for common distributions
We now summarize the PR curves of the same location-scale families as in the ROC case (Table 2), with the same notation: \(X_y=\sigma_yZ_y+\mu_y\) with \(Z_0,Z_1\) i.i.d. with CDF \(F\), \(\Delta=(\mu_1-\mu_0)/\sigma_1\), \(\rho=\sigma_0/\sigma_1\), \(\Phi\), \([x]_0^1\) and \((x)_+\). We additionally assume that \(F\) is continuous and strictly increasing on \(\{z:0<F(z)<1\}\), which holds for every law in the table below. Unlike the ROC curve, the PR curve also depends on the prevalence \(\pi_1\).
Parametrization by recall. Since \(\mathrm{Recall}(t)=\mathrm{TPR}(t)=1-F\bigl(\tfrac{t-\mu_1}{\sigma_1}\bigr)\), the assumption on \(F\) makes \(\mathrm{Recall}\) a continuous, strictly decreasing bijection from \(\{t:0<\mathrm{Recall}(t)<1\}\) onto \((0,1)\). Hence, for every recall level \(r\in(0,1)\), there is a unique threshold \(t_r\) with \(\mathrm{Recall}(t_r)=r\), and the PR curve can be parametrized by \(r\). We define \[
\begin{align*}
G(r)
& \triangleq \mathrm{FPR}(t_r)\\
& =\mathbb{P}(X_0\ge t_r).
\end{align*}
\] Thus \(G\) expresses the ROC curve in terms of recall: it gives the false-positive rate required to attain a given true-positive rate. Like the ROC curve, \(G\) depends only on the class-conditional distributions, not on the prevalence \(\pi_1\).
The endpoints \(r=0\) and \(r=1\) are excluded on purpose, because uniqueness may fail there. For instance, for the Uniform and Exponential laws, \(\mathrm{Recall}(t)=1\) for all thresholds \(t\) below the support, so there is no single \(t_1\) and the PR curve has a vertical segment at \(\mathrm{Recall}=1\) (see the remark in Appendix B).
Since \(\mathrm{Recall}=\mathrm{TPR}=r\) and \(\mathrm{FPR}=G(r)\), the PR curve is given by
Consequently, the PR curve is determined by \(G\) together with the class prevalence \(\pi_1\). This representation will also be useful for expressing average precision as an integral over recall.
Table 3: FPR as a function of recall, for common distributions. The precision is \(\mathrm{Precision}(r)=\bigl(1+\tfrac{\pi_0}{\pi_1}\tfrac{G(r)}{r}\bigr)^{-1}\).
Where the table comes from. As for ROC, all the rows follow from a single identity (Theorem 19): at the threshold with recall \(r\), \[
G(r)=1-F\Bigl(\frac{F^{-1}(1-r)+\Delta}{\rho}\Bigr),
\] which simplifies to \(G(r)=F\bigl((F^{-1}(r)-\Delta)/\rho\bigr)\) when \(F\) is symmetric (Corollary 4). The statements, the proofs, and the row-by-row computations are collected in Appendix B.
Visualization
The widget below lets you explore Theorem 19 interactively for the distribution described in Table 3. Use the sliders on the left to set \(\mu_0,\sigma_0\) (healthy class), \(\mu_1,\sigma_1\) (sick class) and the Prevalence\(\pi_1\), and drag the Threshold slider to move \(t\).
The middle panel shows the two population densities plotted vertically: \(X_1\) (sick) on the left in blue, \(X_0\) (healthy) on the right in red. The horizontal black line marks the current threshold \(t\); the darker-shaded region of each density is \(\{X<t\}\) (the portion of that class predicted negative under the rule \(\hat{Y}_t=\mathbb{1}\{X\ge t\}\)) while the lighter region above the line is predicted positive.
The right panel shows the theoretical PR curve for the current \((\mu_0,\sigma_0,\mu_1,\sigma_1,\pi_1)\), with the dashed horizontal line \(\mathrm{Precision}=\pi_1\) as the random-guessing reference (Theorem 14) and a marker at the operating point \((\mathrm{Recall}(t),\mathrm{Precision}(t))\) corresponding to the threshold you’ve selected, so you can see exactly which point on the curve a given threshold produces, and watch the curve itself deform as you change the means, standard deviations, their ratio \(\rho=\sigma_0/\sigma_1\), or the prevalence \(\pi_1\). Whatever the parameters, the curve ends at \((1,\pi_1)\) on the right.
The title above the plot reports the current \(\Delta=(\mu_1-\mu_0)/\sigma_1\), \(\rho = \sigma_0/\sigma_1\) and \(\pi_1\).
Figure 5: PR curve for classic distributions
Note: Visually, the recall is the ratio of the light blue area over the full blue area on the left, while the precision is the share of the predicted-positive mass that is blue: \(\pi_1\) times the light blue fraction, divided by that quantity plus \(\pi_0\) times the light red fraction.
Interpretation
\(\Delta=(\mu_1-\mu_0)/\sigma_1\) and \(\rho=\sigma_0/\sigma_1\) set the difficulty and shape of the underlying ROC curve, hence the function \(G\); the prevalence \(\pi_1\) then rescales the odds of precision.
Odds form:\(\dfrac{\mathrm{Precision}}{1-\mathrm{Precision}}=\dfrac{\pi_1}{\pi_0}\cdot\dfrac{\mathrm{TPR}}{\mathrm{FPR}}\). Lowering \(\pi_1\) with the ROC curve fixed lowers the precision at every recall level, and the PR curve is pulled toward the horizontal axis. At fixed \((\Delta,\rho)\) the ROC curve is the same for every \(\pi_1\), but the PR curve is not.
\(\Delta=0,\ \rho=1\) (symmetric \(F\)):\(G(r)=r\), so \(\mathrm{Precision}\equiv\pi_1\): the PR curve is the random-guessing baseline of Theorem 14. As in the ROC case, \(\Delta=0\) alone is not enough when \(\rho\ne1\).
Right end of the curve, \(\mathrm{Recall}\to1\):\(\mathrm{Precision}\to\pi_1\) for every \((\Delta,\rho,F)\), which is the universal endpoint \((1,\pi_1)\).
Left end of the curve, \(\mathrm{Recall}\to0\): the limit is governed by the tail ratio \(G(r)/r\) as \(r\to0\). In the Gaussian case it is \(1\) if \(\rho<1\), or \(\rho=1\) and \(\Delta>0\); it is \(\pi_1\) if \(\rho=1\) and \(\Delta=0\); and it is \(0\) if \(\rho>1\), or \(\rho=1\) and \(\Delta<0\). A more dispersed healthy class (\(\rho>1\)) therefore puts mostly healthy samples in the extreme upper tail.
\(\Delta\to\infty\) (fixed \(\rho\)): \(G(r)\to0\) for each \(r\in(0,1)\), so \(\mathrm{Precision}(r)\to1\) and the curve approaches the ideal curve through \((1,1)\).
Now we illustrate the empirical counterpart, when one samples elements from the distributions but without access to the true densities.
Figure 6: Empirical counterpart of PR curve for classic distributions.
Note: the jittering on the x-axis for the scatter plots is not meaningful but only here for visualization.
Average Precision (AP)
Average Precision (AP) summarizes the precision–recall (PR) curve by integrating precision over recall, just as the area under the ROC curve (AUC) summarizes the ROC curve. Unlike ROC AUC, AP depends on the positive-class prevalence \(\pi_1\); under random guessing, its population value is \(\pi_1\).
With \(t_r\) and \(G(r)=\mathrm{FPR}(t_r)\) as defined in the previous section, the precision at recall level \(r\) is \(\mathrm{Precision}(t_r)=\dfrac{\pi_1 r}{\pi_1 r+\pi_0 G(r)}\).
Theorem 15 (Average precision as the area under the PR curve) Let \(X_0\sim\mathcal{L}(X\mid Y=0)\) and \(X_1\sim\mathcal{L}(X\mid Y=1)\) be independent. Assume that the CDF \(F_1\) of \(X_1\) is continuous and strictly increasing wherever \(0<F_1(t)<1\).
Define average precision by \[
\mathrm{AP}\triangleq \mathbb{E}\bigl[\mathrm{Precision}(X_1)\bigr].
\]
Then AP equals the area under the PR curve: \[
\boxed{
\mathrm{AP}
=
\int_0^1
\frac{\pi_1 r}{\pi_1 r+\pi_0G(r)}
\,dr.
}
\]
▶Proof
Let \(X_1'\) be an independent copy of \(X_1\), and define \[
R\triangleq 1-F_1(X_1).
\] By the probability integral transform, \(R\sim\mathrm{Uniform}(0,1)\). Moreover, since \(X_1\) has a continuous distribution, \[
\begin{aligned}
R
&=\mathbb{P}(X_1'\ge X_1\mid X_1)\\
&=\mathrm{Recall}(X_1).
\end{aligned}
\]
By the definition of \(t_r\) and the strict monotonicity of \(F_1\) on its nontrivial range, \(X_1=t_R\) almost surely. Hence, \[
\begin{aligned}
\mathrm{Precision}(X_1)
&=\frac{\pi_1R}
{\pi_1R+\pi_0\mathbb{P}(X_0\ge X_1)}\\[4pt]
&=\frac{\pi_1R}
{\pi_1R+\pi_0G(R)}.
\end{aligned}
\]
Since \(R>0\) almost surely and \(\pi_1>0\), the denominator is positive almost surely. Taking expectations and using \(R\sim\mathrm{Uniform}(0,1)\) yields \[
\begin{aligned}
\mathrm{AP}
&=\mathbb{E}\bigl[\mathrm{Precision}(X_1)\bigr]\\[4pt]
&=\int_0^1
\frac{\pi_1r}{\pi_1r+\pi_0G(r)}
\,dr\\[4pt]
&=\int_0^1\mathrm{Precision}(t_r)\,dr.
\end{aligned}
\]
The last integral is the area under the PR curve, parametrized by recall. This establishes the result.
Two remarks connect the AP with the ROC curve and the AUC.
Dependence on the prevalence.\(G\) is the ROC curve read in the other direction, so it is fixed by the two conditional laws alone. The prevalence enters only through the weight \(\pi_0/\pi_1\) in \[
\mathrm{Precision}(r)=\Bigl(1+\frac{\pi_0}{\pi_1}\,\frac{G(r)}{r}\Bigr)^{-1}.
\] The AP is therefore a functional of the ROC curve and of \(\pi_1\).
Comparison with the AUC. By Corollary 2, \(\mathrm{AUC}=\mathbb{P}(X_1>X_0)\), hence \(\mathrm{AUC}=\int_0^1\bigl(1-G(r)\bigr)\,dr\), which does not depend on \(\pi_1\). Both metrics average over recall levels \(r\) uniformly, but they penalize the false positive rate differently: the AUC penalizes \(G(r)\) itself, equally at every recall level, whereas the AP penalizes the ratio \(G(r)/r\). A given false positive rate therefore costs more precision at low recall, so the AP rewards good behavior there (see the end of the interpretation above).
Like the AUC, the AP has a natural empirical counterpart that only depends on the ranks of the samples.
Theorem 16 (Empirical average precision) Let \(X_{1,1},\dots,X_{1,n_1}\) and \(X_{0,1},\dots,X_{0,n_0}\) be independent i.i.d. samples of the continuous variables \(X_1\) and \(X_0\), so that there are no ties almost surely. Let \(X_{1,(1)}>\dots>X_{1,(n_1)}\) be the ordered \(X_{1,i}\)’s, and \(\rho_k\) the rank of \(X_{1,(k)}\) from the top of the pooled sample. Then the step-wise area under the empirical PR curve is \[
\begin{align*}
\widehat{\mathrm{AP}}
& =\frac1{n_1}\sum_{k=1}^{n_1}\frac{k}{\rho_k}\\
& =\frac1{n_1}\sum_{i=1}^{n_1}\widehat{\mathrm{Precision}}(X_{1,i}),
\end{align*}
\] the plug-in estimator of \(\mathbb{E}[\mathrm{Precision}(X_1)]\).
▶Proof
Step 1: precision at the \(k\)-th positive sample. Fix \(k\) and let \(t=X_{1,(k)}\). Since there are no ties, the positive samples \(\ge t\) are exactly \(X_{1,(1)},\dots,X_{1,(k)}\), so \(\mathrm{TP}(t)=k\). The pooled samples \(\ge t\) are exactly the \(\rho_k\) samples ranked first to \(\rho_k\)-th from the top, so \(\hat{\mathrm{P}}_t=\rho_k\). Hence \[
\widehat{\mathrm{Precision}}(X_{1,(k)})=\frac{\mathrm{TP}(t)}{\hat{\mathrm{P}}_t}=\frac{k}{\rho_k}.
\]
Step 2: the step-wise area. The recall jumps by \(1/n_1\) exactly when \(t\) reaches a positive sample \(X_{1,(k)}\), and the precision at that threshold is \(k/\rho_k\). The step-wise area is the sum over the jumps of (jump size) \(\times\) (precision at the jump), that is \[
\widehat{\mathrm{AP}}=\sum_{k=1}^{n_1}\frac1{n_1}\cdot\frac{k}{\rho_k},
\] which is the first form.
Step 3: the second form. By Step 1, \(\widehat{\mathrm{AP}}=\frac1{n_1}\sum_{k=1}^{n_1}\widehat{\mathrm{Precision}}(X_{1,(k)})\). The \(X_{1,(k)}\) are the \(X_{1,i}\) listed in decreasing order, so permuting the summands gives \[
\widehat{\mathrm{AP}}=\frac1{n_1}\sum_{i=1}^{n_1}\widehat{\mathrm{Precision}}(X_{1,i}),
\] which is the second form.
Step 4: plug-in. Let \(\hat{\pi}_y=n_y/(n_0+n_1)\). Then \(\hat{\pi}_1\widehat{\mathrm{TPR}}(t)=\mathrm{TP}(t)/(n_0+n_1)\) and \(\hat{\pi}_0\widehat{\mathrm{FPR}}(t)=\mathrm{FP}(t)/(n_0+n_1)\), so the formula of Theorem 11, evaluated on these empirical quantities, gives \[
\begin{align*}
\frac{\hat{\pi}_1\,\widehat{\mathrm{TPR}}(t)}{\hat{\pi}_1\,\widehat{\mathrm{TPR}}(t)+\hat{\pi}_0\,\widehat{\mathrm{FPR}}(t)}
& = \frac{\mathrm{TP}(t)}{\mathrm{TP}(t)+\mathrm{FP}(t)}\\
& = \widehat{\mathrm{Precision}}(t).
\] Since \(\frac1{n_1}\sum_i\delta_{X_{1,i}}\) is the empirical law of \(X_1\), the second form is \(\mathbb{E}[\mathrm{Precision}(X_1)]\) with the law of \(X_1\) replaced by its empirical law, and \(\mathrm{Precision}\) by \(\widehat{\mathrm{Precision}}\).
In Figure 6, the gray curve is the empirical PR curve, with consecutive points joined by straight segments for display only. The \(\widehat{\mathrm{AP}}\) of Theorem 16 is not the area under this drawn curve: it is the area of the staircase with height \(k/\rho_k\) on the \(k\)-th recall step. This is what scikit-learn’s average_precision_score computes. The trapezoidal rule gives a different value, because precision is not linear in the recall between two points. The gap shrinks as \(n_1\) grows.
Finally, we place the AP between two extreme values, which gives it the same kind of scale as the AUC has between \(0\) and \(1\).
Theorem 17 (Range and reference values of the AP) Under the assumptions of Theorem 15, \[
\boxed{1+\frac{\pi_0}{\pi_1}\ln\pi_0\ \le\ \mathrm{AP}\ \le\ 1.}
\] The upper bound is attained if and only if the classifier is perfect (\(\mathrm{AUC}=1\)), and the lower bound if and only if it is always wrong (\(\mathrm{AUC}=0\)). Moreover, for random guessing (\(X_0\stackrel d=X_1\)), \(\mathrm{AP}=\pi_1\), and the lower bound is strictly smaller than \(\pi_1\): an always-wrong classifier does worse than random guessing.
▶Proof
By Theorem 15, \(\mathrm{AP}=\displaystyle\int_0^1\frac{\pi_1 r}{\pi_1 r+\pi_0G(r)}\,dr\), where \(G(r)\in[0,1]\) is an FPR. The integrand is decreasing in \(G(r)\), so both bounds follow from \(0\le G\le 1\).
Upper bound (perfect classifier). With \(G(r)\ge0\), the integrand is at most \(1\), so \(\mathrm{AP}\le1\). Equality holds if and only if \(G=0\) almost everywhere, that is, if and only if \(\int_0^1 G(r)\,dr=0\). Since \(\mathrm{AUC}=\int_0^1(1-G(r))\,dr\), this means \(\mathrm{AUC}=1\): the perfect classifier of Theorem 12.
Lower bound (always-wrong classifier). With \(G(r)\le1\), the integrand is at least \(\frac{\pi_1 r}{\pi_1 r+\pi_0}\), so \[
\mathrm{AP}\ \ge\ \int_0^1\frac{\pi_1 r}{\pi_1 r+\pi_0}\,dr=1+\frac{\pi_0}{\pi_1}\ln\pi_0 .
\] Equality holds if and only if \(G=1\) almost everywhere, that is, if and only if \(\mathrm{AUC}=0\): the always-wrong classifier of Theorem 13, whose PR curve is exactly the arc \(\frac{\pi_1 r}{\pi_1 r+\pi_0}\).
Random guessing. If \(X_0\stackrel d=X_1\), then \(\mathrm{Precision}(t)=\pi_1\) for every \(t\) where it is defined (Theorem 14). Hence \(\mathrm{AP}=\pi_1\).
Comparison. Writing \(a=\pi_1\), the inequality \(1+\frac{\pi_0}{\pi_1}\ln\pi_0<\pi_1\) is equivalent to \(\ln(1-a)<-a\), which holds for every \(a\in(0,1)\).
Reading the AP. The AP is read against its baseline \(\pi_1\): an AP of \(0.3\) is excellent when \(\pi_1=0.01\) but poor when \(\pi_1=0.4\). APs are therefore not comparable across data sets with different prevalence, unlike AUCs.
Back to the COVID-19 use case
Data preprocessing
The file supplementary_dataset2.csv contains the sample-level data of the benchmark study: antibody assay measurements, sample-collection categories, and timing information such as the days since symptom onset.
Our primary benchmark follows the study’s cohort definition. Positives are laboratory-confirmed SARS-CoV-2 samples collected at least 20 days after symptom onset (only the latest qualifying sample is kept per participant), from the Oxford and NHSBT/Porton study populations. Negatives are the pre-pandemic samples of the Oxford Biobank. The preprocessing has three steps:
select the study populations of the primary analysis;
apply the eligibility criteria and keep the latest qualifying sample per participant;
convert the assay measurements to numbers and keep the samples with complete results on all five assays.
The resulting complete-case cohort has \(n=1512\) observations, with \(n_1=536\) positive and \(n_0=976\) negative samples. These counts describe the composition of the benchmark, not the prevalence of infection in a clinical population.
Code
import numpy as npimport pandas as pdimport matplotlib# Colors of the figures: Set1 palette (positives and negatives)colors = matplotlib.colormaps["Set1"].colorsraw_df = pd.read_csv("supplementary_dataset2.csv", comment="#", na_values=["n/a", "NA", "NaN"],)# Drop the samples that are not SARS-CoV-2 cases (other coronaviruses,# other respiratory infections, convalescent or seasonal samples, ...)non_sars_cov2 = raw_df[~raw_df["samplecollection"].eq("obb_negatives")& raw_df["diseaseseverity"].str.contains("MERS|coronavirus|respiratory|convalescent|seasonal|other", case=False, na=False, )]raw_df = raw_df.drop(non_sars_cov2.index).copy()raw_df["covid_status"] = np.where( raw_df["samplecollection"].eq("obb_negatives"), "Negative", "Positive")assay_cols = ["abbott_result","diasorin_result","oxford_result","roche_result","siemens_result",]def assay_to_numeric(x):"""Convert a raw assay result to a number. Censored values are mapped to a finite value: "<3.80" -> 1.90 (half the detection limit), ">10.00" -> 10.00. """if pd.isna(x):return np.nan x =str(x).strip()try:if x.startswith("<"):returnfloat(x[1:].strip()) /2if x.startswith(">"):returnfloat(x[1:].strip())returnfloat(x)exceptValueError:return np.nan# 1. Study populations of the primary benchmarkdf = raw_df[ raw_df["samplecollection"].isin( ["obb_negatives", "oxford_positives", "porton_nhsbt_positives"] )].copy()df["dayssincesymptoms"] = pd.to_numeric(df["dayssincesymptoms"], errors="coerce")# 2. Eligibility# - all pre-pandemic Oxford Biobank samples are negative references;# - Oxford positives require >= 20 days since symptom onset;# - NHSBT positives are known to be >= 28 days post symptom onset.eligible_df = df[ df["covid_status"].eq("Negative")| df["samplecollection"].eq("porton_nhsbt_positives")| ( df["samplecollection"].eq("oxford_positives")& df["dayssincesymptoms"].ge(20) )].copy()# ... and keep the latest qualifying sample per participant# (missing participant IDs must remain distinct; missing delays are NHSBT: 28)eligible_df["participant_id"] = eligible_df["participantnum"].astype("string")missing = eligible_df["participant_id"].isna()eligible_df.loc[missing, "participant_id"] = ("missing_"+ eligible_df.index[missing].astype(str))eligible_df["sort_days"] = eligible_df["dayssincesymptoms"].fillna(28)benchmark_df = ( eligible_df.sort_values(["participant_id", "sort_days"], ascending=[True, False]) .drop_duplicates("participant_id", keep="first") .drop(columns=["participant_id", "sort_days"]) .copy())# 3. Numeric assay values; complete cases on the five assaysfor col in assay_cols: benchmark_df[col] = benchmark_df[col].map(assay_to_numeric)# Zero values cannot be shown on a log axis: use a small positive valuebenchmark_df["abbott_result"] = benchmark_df["abbott_result"].replace(0, 0.005)pcr_df = benchmark_df.dropna(subset=assay_cols).copy()print("Complete-case cohort:")print(pcr_df["covid_status"].value_counts())print("\nBy sample collection:")print(pcr_df.groupby(["covid_status", "samplecollection"]).size())# Assay labels and manufacturers' recommended thresholds (None: not available)dict_results_inv = {"abbott_result": "Abbott","diasorin_result": "DiaSorin","oxford_result": "Oxford","roche_result": "Roche","siemens_result": "Siemens",}dict_threshold = {"abbott_result": 1.4,"diasorin_result": 15.0,"oxford_result": None,"roche_result": 1.0,"siemens_result": 1.0,}
The interactive figure below runs client-side (Plotly.js), without a live Python kernel. We therefore export to JSON the sample values, the jitter offsets, the colors, and the ROC and PR curves of each assay, computed with the roc and precision_recall functions above.
Code
import jsonimport matplotlib.colors as mcolorsfrom sklearn.metrics import auc, average_precision_scoredef export_roc_pr_data( df, dict_results_inv, dict_threshold, colors, seed=123, jitter=0.3, n_thresh=200):"""ROC / PR curves of each assay (on its non-missing observations).""" rng = np.random.RandomState(seed) df_pos_all = df[df["covid_status"].eq("Positive")] df_neg_all = df[df["covid_status"].eq("Negative")]# Horizontal jitter of the scatter panel (purely visual) x_pos = (jitter * (rng.rand(len(df_pos_all)) -0.5)).tolist() x_neg = (0.6+ jitter * (rng.rand(len(df_neg_all)) -0.5)).tolist() assays = {}for key, label in dict_results_inv.items(): dfa = df.loc[df[key].notna()] y_true = dfa["covid_status"].eq("Positive").values score = dfa[key].astype(float).values# Threshold range, valid on a log scale positive_scores = score[score >0] lower = positive_scores.min() /2iflen(positive_scores) else1e-6 upper = score.max() *2if score.max() >0else1.0if upper <= lower: upper = lower *2 ylims = [float(max(lower, 1e-12)), float(upper)] thresholds = np.logspace(np.log10(ylims[0]), np.log10(ylims[1]), num=n_thresh) tpr, fpr = roc(thresholds, y_true, score) precision, recall = precision_recall(thresholds, y_true, score)# AUC: trapezoidal rule on the ROC curve; AP: scikit-learn's# step-wise definition (see the AP section above) order = np.argsort(fpr) auc_value =float(auc(np.array(fpr)[order], np.array(tpr)[order])) ap_value =float(average_precision_score(y_true, score)) thr = dict_threshold[key] assays[label] = {"key": key,"pos": dfa.loc[y_true, key].astype(float).tolist(),"neg": dfa.loc[~y_true, key].astype(float).tolist(),"ylims": ylims,"threshold_init": float(np.median(score)),"threshold_reco": Noneif thr isNoneelsefloat(thr),"roc_fpr": fpr,"roc_tpr": tpr,"auc": auc_value,"pr_precision": precision,"pr_recall": recall,"ap": ap_value,"n_pos": int(y_true.sum()),"n_neg": int((~y_true).sum()), }# Best assay (largest AUC) first assays =dict(sorted(assays.items(), key=lambda kv: kv[1]["auc"], reverse=True))return {"x_pos": x_pos,"x_neg": x_neg,"colors": {"pos": mcolors.to_hex(colors[1]), "neg": mcolors.to_hex(colors[0])},"assays": assays,"default_assay": "DiaSorin"if"DiaSorin"in assays elsenext(iter(assays), None),"n_pos": int(len(df_pos_all)),"n_neg": int(len(df_neg_all)), }def subsample_positives(df, frac_pos=0.05, seed=123):"""Keep every negative sample, but only a fraction of the positives.""" pos = df[df["covid_status"].eq("Positive")] neg = df[df["covid_status"].eq("Negative")] pos_sub = pos.sample(n=max(1, round(len(pos) * frac_pos)), random_state=seed)return pd.concat([pos_sub, neg], ignore_index=True)# Full data, and a teaching example with a much rarer positive classfor filename, data in [ ("roc_data.json", pcr_df), ("roc_data_imbalanced.json", subsample_positives(pcr_df, frac_pos=0.05)),]: export = export_roc_pr_data(data, dict_results_inv, dict_threshold, colors)withopen(filename, "w") as f: json.dump(export, f)
Interactive exploration: assay results, ROC curve and precision/recall curve
The widget below compares several assays at once. Tick one or more of them in the left-hand box: the first one you tick is the primary assay, and the threshold slider (log scale) acts on it. The three panels show:
left: the individual results of the primary assay, split by true status;
top right: the ROC curves of all the selected assays;
bottom right: their precision/recall curves, with the AP (Average Precision) in the legend.
When the manufacturer provides a recommended threshold, it appears as a dotted grey line (left panel) and a grey cross on the curves (“Reco. point”).
Assay (click)
Threshold
Figure 7: Assays on the full data (536 positive and 976 negative samples).
Now we rerun the same assays, with the same thresholds, but keep only 5% of the positive samples (the negatives are untouched). For any given assay, the ROC curve barely moves, while the precision/recall curve and the AP drop markedly. This is what the ray-slope argument above predicts: shrinking \(\mathrm{P}\) raises the ratio \(\mathrm{N}/\mathrm{P}\), which pushes the precision down at every recall level, while the ray slopes, hence the ROC curve, stay the same.
Assay (click)
Threshold
Figure 8: Same assays, with only 5% of the positive samples kept (same panels and thresholds as in Figure 7).
References
Ainsworth, Mark, Monique Andersson, Kathryn Auckland, and The National SARS-CoV-2 Serology Assay Evaluation Group. 2020. “Performance Characteristics of Five Immunoassays for SARS-CoV-2: A Head-to-Head Benchmark Comparison.”The Lancet Infectious Diseases 20 (12): 1390–400. https://doi.org/10.1016/S1473-3099(20)30634-4.
Chen, Wenyu, Chen Miao, Zhenghao Zhang, Cathy Sin-Hang Fung, Ran Wang, Yizhen Chen, Yan Qian, et al. 2024. “Commonly Used Software Tools Produce Conflicting and Overly-Optimistic AUPRC Values.”Genome Biology 25 (1): 118. https://doi.org/10.1186/s13059-024-03266-y.
Kent, Allen, Madeline Berry, Fred U. Luehrs, and James W. Perry. 1955. “Machine Literature Searching VIII. Operational Criteria for Designing Information Retrieval Systems.”American Documentation 6 (2): 93–101.
Peterson, W. W., T. G. Birdsall, and W. C. Fox. 1954. “The Theory of Signal Detectability.”Transactions of the IRE Professional Group on Information Theory 4 (4): 171–212.
Appendix A: ROC curves for location-scale families
This appendix gathers the computations behind Table 2, with the notation of the main text: \(X_y=\sigma_yZ_y+\mu_y\), where \(Z_0,Z_1\) are i.i.d. with CDF \(F\), \(\Delta=(\mu_1-\mu_0)/\sigma_1\), \(\rho=\sigma_0/\sigma_1\), \(\Phi\), \([x]_0^1\) and \((x)_+\).
ROC curve for location-scale family
Theorem 18 (Location-scale ROC curve) Let \(Z_0,Z_1\) be i.i.d. real-valued random variables with common CDF \(F\), continuous and strictly increasing on \(\{z\in\mathbb{R}:0<F(z)<1\}\), so that \(F^{-1}:(0,1)\to\mathbb{R}\) is well defined. Fix \(\mu_0,\mu_1\in\mathbb{R}\), \(\sigma_0,\sigma_1>0\), and let \[
X_0\triangleq \sigma_0Z_0+\mu_0, \quad X_1\triangleq \sigma_1Z_1+\mu_1
\] be independent. Writing \(\Delta\triangleq (\mu_1-\mu_0)/\sigma_1\), \(\rho\triangleq \sigma_0/\sigma_1\), for every \(\mathrm{FPR}\in (0,1)\),
where the middle step uses symmetry once more, at the point \(-\rho F^{-1}(\mathrm{FPR})-\Delta\).
Applications: the symmetric rows
Each of the following is an immediate instance of Corollary 3: verify that \(F\) is symmetric, write down \(F^{-1}\), and substitute.
Gaussian. With \(\Phi\) the standard normal CDF, \[
\Phi(x)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{x}e^{-u^2/2}\,du,
\]\(F=\Phi\) satisfies \(\Phi(-x)=1-\Phi(x)\) by construction. Substituting in Corollary 3 gives \[
\mathrm{TPR}=\Phi\bigl(\Delta+\rho\,\Phi^{-1}(\mathrm{FPR})\bigr).
\]
Cauchy. The standard Cauchy CDF is \(F(x)=\tfrac12+\tfrac1\pi\tan^{-1} x\). Since \(\tan^{-1}\) is odd, \(F(-x)=1-F(x)\). Solving \[
\tfrac12+\tfrac1\pi\tan^{-1}(F^{-1}(p))=p
\] gives \[
F^{-1}(p)=\tan\bigl(\pi(p-\tfrac12)\bigr).
\] Substituting gives the Cauchy entry in the Table 2.
Logistic. The logistic CDF is \(F(x)=\dfrac{1}{1+e^{-x}}\). It is symmetric since \(F(-x)=1-F(x)\), and \(F^{-1}(p)=\ln\dfrac{p}{1-p}\). Therefore, \[
\begin{align*}
\mathrm{TPR}
& = F\Bigl(\Delta+\rho\ln\tfrac{\mathrm{FPR}}{1-\mathrm{FPR}}\Bigr)\\
& = \frac{1}{1+e^{-\Delta}\left(\tfrac{1-\mathrm{FPR}}{\mathrm{FPR}}\right)^\rho}.
\end{align*}
\] Multiplying numerator and denominator by \(e^\Delta\mathrm{FPR}^\rho\) gives the entry in Table 2.
Uniform \((-\tfrac12,\tfrac12)\). The CDF is obtained by clipping the shifted variate, \[
F(z)=\bigl[z+\tfrac12\bigr]_0^1,
\] which is symmetric on \(\mathbb{R}\), with \(F^{-1}(p)=p-\tfrac12\) on \((0,1)\).
By the theorem, \(\mathrm{TPR}=F(z_1)\) where \(z_1=\Delta+\rho z_0\) and \(z_0=F^{-1}(\mathrm{FPR})=\mathrm{FPR}-\tfrac12\). Hence \[
\begin{align*}
\mathrm{TPR}
& = F\bigl(\Delta+\rho\bigl(\mathrm{FPR}-\tfrac12\bigr)\bigr)\\
& = \Bigl[\Delta+\tfrac{1-\rho}{2}+\rho\,\mathrm{FPR}\Bigr]_0^1 .
\end{align*}
\]
The remaining rows: Gumbel and Exponential (not symmetric)
Gumbel. The Gumbel-max CDF is \(F(x)=\exp(-e^{-x})\). Its inverse is \(F^{-1}(p)=-\ln(-\ln p)\), so with the previous theorem, \(1-\mathrm{TPR}=F\left(\rho \ln\left(\frac{1}{-\ln(1-\mathrm{FPR})}\right) - \Delta\right)\).
Setting \(w\triangleq -\ln(1-\mathrm{FPR})>0\), the argument of \(F\) is \(-\rho\ln w-\Delta\), and \[
\begin{align*}
F(-\rho\ln w-\Delta)
& = \exp\bigl(-e^{\rho\ln w+\Delta}\bigr)\\
& = \exp\bigl(-e^\Delta w^\rho\bigr),
\end{align*}
\] hence \[
\mathrm{TPR}(\mathrm{FPR})=1-e^{\Bigl(-e^\Delta\bigl(-\ln(1-\mathrm{FPR})\bigr)^\rho\Bigr)}.
\]
Exponential. The exponential CDF is \(F(z)=(1-e^{-z})_+\). Its inverse on \((0,1)\) is \(F^{-1}(p)=-\ln(1-p)\), so, with Theorem 18, \[
\mathrm{TPR}(t)=1-F\bigl(\rho\,(-\ln\mathrm{FPR})-\Delta\bigr).
\] When \(\rho(-\ln\mathrm{FPR})-\Delta\ge0\) (i.e. \(\mathrm{FPR}\le e^{-\Delta/\rho}\)), \(F(x)=1-e^{-x}\) gives \(1-F(\cdot)=e^{-(\rho(-\ln\mathrm{FPR})-\Delta)}=e^\Delta\mathrm{FPR}^\rho\); otherwise the argument is negative, \(F\equiv0\) there, and \(\mathrm{TPR}=1\). Hence \[
\boxed{\mathrm{TPR}(\mathrm{FPR})=\min\bigl(e^\Delta\mathrm{FPR}^\rho,\,1\bigr).}
\]
Appendix B: PR curves for location-scale families
This appendix gathers the computations behind Table 3, with the notation of Appendix A, the prevalence \(\pi_1\), and \(G(r)=\mathrm{FPR}(t_r)\) as defined in the main text.
PR curve for location-scale family
Theorem 19 (Location-scale PR curve) Under the hypotheses of Theorem 18, let \(\pi_1\in(0,1)\) and \(\pi_0=1-\pi_1\). Then for every \(r\in(0,1)\) there is a unique threshold \(t_r\) with \(\mathrm{Recall}(t_r)=r\), and \[
\begin{align*}
\mathrm{FPR}(t_r)
& = 1-F\Bigl(\tfrac{F^{-1}(1-r)+\Delta}{\rho}\Bigr)\\
\mathrm{Precision}(t_r)
& = \frac{\pi_1\,r}{\pi_1\,r+\pi_0\,G(r)}
\end{align*}
\]
▶Proof
Threshold. Since \(\mathrm{Recall}(t)=\mathrm{TPR}(t)=1-F\bigl(\tfrac{t-\mu_1}{\sigma_1}\bigr)\), the equation \(\mathrm{Recall}(t)=r\) reads \(F\bigl(\tfrac{t-\mu_1}{\sigma_1}\bigr)=1-r\in(0,1)\). As \(F\) is strictly increasing on \(\{0<F<1\}\), its unique solution is \[
t_r=\mu_1+\sigma_1F^{-1}(1-r).
\]
FPR. At \(t_r\), \[
\begin{align*}
\frac{t_r-\mu_0}{\sigma_0}
& =
\frac{\sigma_1F^{-1}(1-r)+\mu_1-\mu_0}{\sigma_0}\\
& = \frac{F^{-1}(1-r)+\Delta}{\rho},
\end{align*}
\] so \(\mathrm{FPR}(t_r)=1-F\bigl(\tfrac{t_r-\mu_0}{\sigma_0}\bigr)=G(r)\).
Precision. By definition, \[
\mathrm{Precision}(t)=\frac{\pi_1\,\mathrm{TPR}(t)}{\pi_1\,\mathrm{TPR}(t)+\pi_0\,\mathrm{FPR}(t)},
\] whose denominator is at least \(\pi_1 r>0\) at \(t=t_r\). Substituting \(\mathrm{TPR}(t_r)=r\) and \(\mathrm{FPR}(t_r)=G(r)\) gives the second equality.
Corollary: symmetric location-scale PR
Corollary 4 (Symmetric location-scale PR curve) Under the hypotheses of Theorem 19, suppose in addition that \(F(-x)=1-F(x)\) for every \(x\in\mathbb{R}\). Then for every \(r\in(0,1)\), \[
\boxed{
G(r)=F\Bigl(\frac{F^{-1}(r)-\Delta}{\rho}\Bigr).
}
\]
▶Proof
Fix \(r\in(0,1)\) and let \(z\triangleq F^{-1}(r)\), so that \(0<F(z)=r<1\).
Step 1: \(F^{-1}(1-r)=-z\). By symmetry, \(F(-z)=1-F(z)=1-r\in(0,1)\). Hence \(-z\) lies in \(\{0<F<1\}\), where \(F\) is strictly increasing, and \(-z\) is the unique point of this set with \(F=1-r\). That is, \(F^{-1}(1-r)=-z\).
Step 2: simplify \(G\). Put \(x\triangleq\dfrac{z-\Delta}{\rho}\). By Step 1, the argument of \(F\) in the definition of \(G(r)\) is \[
\frac{F^{-1}(1-r)+\Delta}{\rho}=\frac{-z+\Delta}{\rho}=-x.
\] Applying symmetry at the point \(x\), \[
\begin{align*}
G(r)
& = 1-F(-x)\\
& = 1-\bigl(1-F(x)\bigr)
& = F(x)\\
& = F\Bigl(\frac{F^{-1}(r)-\Delta}{\rho}\Bigr).
\end{align*}
\]
Applications: the symmetric rows
Each is an immediate instance of Corollary 4: verify that \(F\) is symmetric, write down \(F^{-1}\), and substitute into \(G\).
Gaussian.\(F=\Phi\) satisfies \(\Phi(-x)=1-\Phi(x)\), so \[
G(r)=\Phi\Bigl(\frac{\Phi^{-1}(r)-\Delta}{\rho}\Bigr).
\]
Cauchy.\(F(x)=\tfrac12+\tfrac1\pi\tan^{-1}x\) is symmetric since \(\tan^{-1}\) is odd, with \(F^{-1}(p)=\tan\bigl(\pi(p-\tfrac12)\bigr)\). Hence \(G(r)=\tfrac12+\tfrac1\pi\tan^{-1}\!\Bigl(\frac{\tan(\pi(r-\frac12))-\Delta}{\rho}\Bigr).\)
Logistic.\(F(x)=\dfrac{1}{1+e^{-x}}\) is symmetric, with \(F^{-1}(p)=\ln\dfrac{p}{1-p}\). Therefore \[
\begin{aligned}
G(r)
&=\frac{1}{1+\exp\bigl(-\tfrac{1}{\rho}\bigl(\ln\tfrac{r}{1-r}-\Delta\bigr)\bigr)}\\
&=\frac{1}{1+e^{\Delta/\rho}\bigl(\tfrac{1-r}{r}\bigr)^{1/\rho}}\\
&=\frac{r^{1/\rho}}{r^{1/\rho}+e^{\Delta/\rho}(1-r)^{1/\rho}}.
\end{aligned}
\]
Uniform \((-\tfrac12,\tfrac12)\).\(F(z)=[z+\tfrac12]_0^1\) is symmetric, with \(F^{-1}(p)=p-\tfrac12\) on \((0,1)\). Hence \[
\begin{align*}
G(r)
& = F\Bigl(\frac{r-\frac12-\Delta}{\rho}\Bigr)\\
& = \Bigl[\frac{r-\Delta-\frac{1-\rho}{2}}{\rho}\Bigr]_0^1 .
\end{align*}
\]
The remaining rows: Gumbel and Exponential (not symmetric)
Gumbel.\(F(x)=\exp(-e^{-x})\) and \(F^{-1}(p)=-\ln(-\ln p)\), so \(F^{-1}(1-r)=-\ln(-\ln(1-r))\). The argument of \(F\) in \(G(r)\) is \(a\triangleq\tfrac{\Delta-\ln(-\ln(1-r))}{\rho}\), and \[
\begin{align*}
F(a)
& = \exp\bigl(-e^{-a}\bigr)\\
& =
e^{\Bigl(-\bigl(e^{-\Delta}(-\ln(1-r))\bigr)^{1/\rho}\Bigr)}.
\end{align*}
\] Hence, \[
G(r)=1-e^{\Bigl(-e^{-\Delta/\rho}\bigl(-\ln(1-r)\bigr)^{1/\rho}\Bigr)}.
\]
Exponential.\(F(z)=(1-e^{-z})_+\) and \(F^{-1}(p)=-\ln(1-p)\), so \(F^{-1}(1-r)=-\ln r\) and the argument of \(F\) is \(a\triangleq\tfrac{\Delta-\ln r}{\rho}\). If \(a\ge0\), then \(1-F(a)=e^{-a}=e^{-\Delta/\rho}r^{1/\rho}\). If \(a<0\), then \(F(a)=0\) and \(G(r)=1\). Hence \[
\boxed{G(r)=\min\bigl(e^{-\Delta/\rho}\,r^{1/\rho},\,1\bigr).}
\]
Remark (bounded-below supports). For the Uniform and Exponential laws, recall \(1\) can be reached at a finite threshold, so the plotted curve \(r\mapsto(r,\mathrm{Precision}(r))\) stops before \(r=1\). The remaining part of the PR curve is the vertical segment at \(\mathrm{Recall}=1\), from \(\mathrm{Precision}(1^-)\) down to the universal endpoint \((1,\pi_1)\) (reached as \(t\to-\infty\)).
Citation
BibTeX citation:
@online{salmon2026,
author = {Salmon, Joseph},
title = {ROC {Curve} and {Precision-Recall}},
date = {2026-10-26},
url = {https://josephsalmon.eu/blog/roc-curve/},
langid = {en}
}
Source Code
---title: "ROC Curve and Precision-Recall"author: - name: Joseph Salmon url: https://josephsalmon.eu/ orcid: 0000-0002-3181-0634date: "2026-10-26"description: "We Will, We Will ROC You: a long story on ROC and Precision-Recall curves."categories: [evaluation, risk, precision, recall, classification, machine learning]image: "roc-curve.png"image-alt: "Receiver operating characteristic (ROC) curves on the right part of the image, in the middle density distribution for the two class conditional distributions, left part widgets controlers."draft: falsereference-location: documentfig-cap-location: bottombibliography: references.bibappendix-cite-as: bibtexresources: - "roc_data.json" - "synthetic_roc_data.json" - "synthetic_pr_data.json" - "roc_data_imbalanced.json" - "*.js"filters: - pseudocodejupyter: python3format: html: table-striped: true from: markdown+emoji html-math-method: mathjax code-tools: true include-in-header: - text: | <script src="https://josephsalmon.eu/blog/proof-toggle.js"></script> <script>window.backupDefineForPlotly = window.define; window.define = undefined;</script> <script src="https://cdn.plot.ly/plotly-2.27.0.min.js"></script> <script>window.define = window.backupDefineForPlotly; window.backupDefineForPlotly = undefined;</script>---::: {.content-visible when-format="html"}::: {.hidden}{{\usepackage{amssymb, amsmath, amsthm}\usepackage{stmaryrd}\DeclareMathOperator*{\argmin}{arg\,min} % thin space, limits underneath in displays\DeclareMathOperator*{\argmax}{arg\,max} % thin space, limits underneath in displays\DeclareMathOperator{\diag}{diag}}}::::::## Introduction and data usedWe illustrate in this post the notion of **ROC curves** and **Precision-Recall curves** using both simple distributions (like Gaussian or uniform) but also data from a head-to-head benchmark evaluation of five SARS-CoV-2 serological assays [@ainsworth2020performance]. We focus only on binary classification in this post.For the empirical part, the underlying study compared the Abbott, DiaSorin, Oxford, Roche, and Siemensassays using a common set of well-characterized samples.We follow @ainsworth2020performance and have used a dataset that contains two groups of samples:- **976 pre-pandemic samples**, used as the known-negative reference group;- **536 laboratory-confirmed SARS-CoV-2-positive samples**, collected at least 20 days after symptom onset.These groups contain a total of$$n = 976 + 536 = 1512$$observations. The proportion of positive samples in this dataset is therefore$$\hat{\pi}_1 = \frac{536}{1512} \approx 0.355,$$or approximately $35.5\%$.**An important distinction:** this proportion describes the composition of the dataset, not necessarily the prevalence of SARS-CoV-2 infection in the population where a diagnostic test would be used. The sample composition may reflect how the study was designed and how samples were selected.This distinction will become important when interpreting precision–recall curves and average precision, which depend on the prevalence of positive cases.The dataset (`supplementary_dataset2.csv`) is available[here](https://figshare.com/articles/dataset/Dataset_Head-to-head_benchmark_evaluation_of_the_sensitivity_and_specificity_of_five_immunoassays_for_SARS-CoV-2_serology_on_1500_samples/12622172).The supplementary material of the associated paper describes the preprocessing used in this analysis.### Diagnostic testing as hypothesis testingA diagnostic test can be framed as a classical **statistical hypothesis test**. In this medical context, we consider the null hypothesis$$\mathcal{H}_0: \text{``you are healthy''} $$against the alternative:$$\mathcal{H}_1: \text{``you are sick''}.$$The presumption of health plays the same role as the *presumption of innocence* in a courtroom: we only reject $\mathcal{H}_0$ (*i.e.,* declare someone positive) when the evidence (here the assay result) is strong enough. Let $Y \in \{0,1\}$ denote the true health status, with $Y=0$ under $\mathcal{H}_0$ (healthy) and $Y=1$ under $\mathcal{H}_1$ (sick). In practice, declaring someone positive means the measured antibody level, modeled by a random variable $X$, must cross an assay-specific diagnostic threshold (cut-off) $t$. The resulting decision rule, our estimate of $Y$ at threshold $t$, is written $\hat{Y}_t$:$$\hat{Y}_t(X) = \mathbb{1}\{X \ge t\}=\begin{cases}1, & X \ge t,\\0, & X < t .\end{cases}$$When no ambiguity is possible we simply write $\hat{Y}_t$.Throughout this post, $\hat{Y}_t$ denotes this rule, except in the symmetry sections where the reversed rule $\mathbb{1}\{X<t\}$ is considered explicitly.It is convenient to isolate the two conditional laws of $X$ given the health status. Under $\mathcal H_0$, $X$ follows $\mathcal{L}(X\mid Y=0)$; under $\mathcal H_1$, it follows $\mathcal{L}(X\mid Y=1)$. We write$$\begin{align*}X_0 & \sim \mathcal{L}(X\mid Y=0), \\X_1 & \sim \mathcal{L}(X\mid Y=1)\end{align*}$$for two independent random variables with these laws. This turns any probability conditional on $Y$ into an ordinary, unconditional probability: for every threshold $t$,$$\begin{align*}\mathbb{P}(X \ge t \mid Y=0) & = \mathbb{P}(X_0\ge t),\\\mathbb{P}(X \ge t \mid Y=1) & = \mathbb{P}(X_1\ge t),\end{align*}$$and we use the right-hand sides from here on, since they involve a single random variable rather than a conditional statement.As in any hypothesis test, two types of errors can occur: rejecting $\mathcal{H}_0$ when it is in fact true (**Type I error**), or failing to reject it when it is in fact false (**Type II error**). In terms of $X_0$ and $X_1$:$$\begin{align*}\mathbb{P}(\text{Type I error}) & = \mathbb{P}(\hat{Y}_t=1 \mid Y=0),\\% = \mathbb{P}(X_0\ge t),\\\mathbb{P}(\text{Type II error}) & = \mathbb{P}(\hat{Y}_t=0 \mid Y=1).% = \mathbb{P}(X_1 < t).\end{align*}$$In words: a Type I error is a **false positive** (the test flags a healthy person), while a Type II error is a **false negative** (the test misses a sick person). The threshold $t$ controls the trade-off between the two: lowering $t$ catches more true cases but triggers more false alarms, and vice versa. **ROC** and **precision/recall curves**, introduced below, are the tools used to visualize this trade-off as $t$ varies.### Naming and key quantitiesThe dataset consists of $n$ observations, denoted $\{(Y_i, X_i)\}_{i=1,\dots,n}$, where $Y_i$ represents the true health status of individual $i$.Naming follows the classical terminology (see *e.g.,* the [Wikipedia article on ROC curves](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)). Given a fixed threshold, each sample falls into exactly one of the four categories below:- **True Positive**: sick and correctly declared positive$$\displaystyle\mathrm{TP} = \sum_{i=1}^n \mathbb{1}\{Y_i=1,\ \hat{Y}_{t,i}=1\}$$- **True Negative**: healthy and correctly declared negative$$\displaystyle\mathrm{TN} = \sum_{i=1}^n \mathbb{1}\{Y_i=0,\ \hat{Y}_{t,i}=0\}$$- **False Positive** (Type I error): healthy but wrongly declared positive$$\displaystyle\mathrm{FP} = \sum_{i=1}^n \mathbb{1}\{Y_i=0,\ \hat{Y}_{t,i}=1\}$$- **False Negative** (Type II error): sick but wrongly declared negative$$\displaystyle\mathrm{FN} = \sum_{i=1}^n \mathbb{1}\{Y_i=1,\ \hat{Y}_{t,i}=0\}$$ From the observed true labels, we define the two class counts:- $n_1=\mathrm{P} = \displaystyle\sum_{i=1}^n \mathbb{1}\{Y_i = 1\} = \mathrm{TP} + \mathrm{FN}$: number of positive (sick) individuals in the sample,- $n_0=\mathrm{N} = \displaystyle\sum_{i=1}^n \mathbb{1}\{Y_i = 0\} = \mathrm{TN} + \mathrm{FP}$: number of negative (healthy) individuals in the sample,and, similarly, from the predicted status $\hat{Y}_t$:- $\hat{\mathrm{P}}_t = \sum_{i=1}^n \mathbb{1}\{\hat{Y}_{t,i} = 1\} = \mathrm{TP} + \mathrm{FP}$: total number of individuals predicted sick at threshold $t$ (**predicted positive**),- $\hat{\mathrm{N}}_t = \sum_{i=1}^n \mathbb{1}\{\hat{Y}_{t,i} = 0\} = \mathrm{TN} + \mathrm{FN}$: total number of individuals predicted healthy at threshold $t$ (**predicted negative**).{fig-alt="Illustration of TP, TN, FP, FN counts on a Covid-19 test example"}**Note**: we will reuse the same color code below, with red for $\mathrm{N}$, blue for $\mathrm{P}$, crosses for a negative test result, and dots for a positive test result.We also define the following rates, used throughout this post. Let $\pi_1\triangleq\mathbb{P}(Y=1)$ and $\pi_0\triangleq\mathbb{P}(Y=0)=1-\pi_1$ denote the class probabilities; we refer to $\pi_1$ as the **prevalence** of the disease in the population.**The true rates.** These are probabilities of the underlying random variables (population quantities, not counts):- $\mathrm{TPR}(t) \triangleq \mathbb{P}\!\left(\hat{Y}_t=1 \mid Y=1\right)$ (**True Positive Rate**, a.k.a. **sensitivity** or **recall**): among sick people, the proportion correctly detected. This is also $\mathbb{P}(X_1\ge t)$.- $\mathrm{TNR}(t) \triangleq \mathbb{P}\!\left(\hat{Y}_t=0 \mid Y=0\right)$ (**True Negative Rate**, a.k.a. **specificity**): among healthy people, the proportion correctly cleared. This is also $\mathbb{P}(X_0< t)$.- $\mathrm{FPR}(t) \triangleq \mathbb{P}\!\left(\hat{Y}_t=1 \mid Y=0\right)$ (**False Positive Rate**): among healthy people, the proportion wrongly flagged, exactly the Type I error probability. This is also $\mathbb{P}(X_0\ge t)$.- $\mathrm{Precision}(t) \triangleq \mathbb{P}\!\left(Y=1 \mid \hat{Y}_t=1\right)$: among people declared positive, the proportion actually sick.The first three depend only on the conditional laws $X_0,X_1$. Precision does not: it also depends on the class probabilities $\pi_0,\pi_1$, and so, unlike the other three, cannot be written purely in terms of $X_0$ and $X_1$ (its exact form is given in @thm-precision-formula below).**Estimators.** In practice $t$, $\pi_1$, and the laws of $X_0,X_1$ are unknown, and each rate above is estimated from a labeled sample $(X_i,Y_i)_{i=1}^n$. With the notation defined earlier, the naive (plug-in) estimator of each rate simply replaces the expectation defining it by the corresponding empirical mean over the sample:$$\begin{align*}\widehat{\mathrm{TPR}}(t) & =\frac{\mathrm{TP}(t)}{\mathrm{P}}, \\\widehat{\mathrm{TNR}}(t) & =\frac{\mathrm{TN}(t)}{\mathrm{N}}, \\\widehat{\mathrm{FPR}}(t) & =\frac{\mathrm{FP}(t)}{\mathrm{N}}, \\\widehat{\mathrm{Precision}}(t) & =\frac{\mathrm{TP}(t)}{\hat{\mathrm{P}}_t}.\\\end{align*}$$## The ROC curveThe **receiver operating characteristic** (ROC) curve comes from **signal detection theory**, which emerged from radar engineering during World War II and was formalised by @peterson1954theory. The problem was to decide, from a noisy measurement, whether a target was present or only background noise. A receiver declares "signal" when the measurement exceeds a threshold, and the ROC curve records how the detection rate and the false-alarm rate move together as that threshold varies. This is exactly the trade-off between catching true threats and raising false alarms. In the language of this post, it is the curve of the power of a threshold test against its Type I error.:::: {#def-roc}# ROC curveFor a threshold $t\in\mathbb{R}$, the **ROC operating point** is the pair of false and true positive rates,$$\begin{align*}\mathrm{FPR}(t) & = \mathbb{P}(X_0\ge t),\\\mathrm{TPR}(t) & = \mathbb{P}(X_1\ge t).\end{align*}$$The **ROC curve** is the parametric curve traced by the operating point as $t$ varies over $\mathbb{R}$:$$\{(\mathrm{FPR}(t), \mathrm{TPR}(t)) : t \in \mathbb{R}\}.$$::::**Note**: Since FPR and TPR are probabilities, the ROC curve is contained in the unit square $[0,1] \times [0,1]$.### Properties#### ROC endpoints:::: {#lem-roc-endpoints}# ROC endpointsAssume that $X_0$ and $X_1$ are almost surely finite. Then the ROC curve has the limiting endpoints:$$\boxed{\lim_{t\to+\infty}\bigl(\mathrm{FPR}(t),\mathrm{TPR}(t)\bigr)=(0,0)}$$and$$\boxed{\lim_{t\to-\infty}\bigl(\mathrm{FPR}(t),\mathrm{TPR}(t)\bigr)=(1,1).}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('lem-roc-endpoints')"}[▶]{#proof-arrow-lem-roc-endpoints .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-lem-roc-endpoints .proof-content}**Step 1: limit as $t\to+\infty$.** By definition,$$\mathrm{FPR}(t) = \mathbb{P}(X_0\ge t), \quad \mathrm{TPR}(t) = \mathbb{P}(X_1\ge t).$$Since $X_0$ and $X_1$ are almost surely finite, $\mathbb{P}(X_y\ge t)\to0$ as $t\to+\infty$ for $y\in\{0,1\}$, giving$$\boxed{\lim_{t\to+\infty} \bigl(\mathrm{FPR}(t),\mathrm{TPR}(t)\bigr) = (0,0).}$$**Step 2: limit as $t\to-\infty$.** Similarly, $\mathbb{P}(X_y\ge t)\to1$ as $t\to-\infty$ for $y\in\{0,1\}$, giving$$\boxed{\lim_{t\to-\infty} \bigl(\mathrm{FPR}(t),\mathrm{TPR}(t)\bigr) = (1,1).}$$:::::::#### Extreme cases##### ROC curve for a perfect classifier:::: {#thm-perfect-separation}# ROC curve for a perfect classifierSuppose there exists a threshold $t^\star \in \mathbb{R}$ such that$$\mathbb{P}(X_0 \ge t^\star) = 0, \quad \mathbb{P}(X_1 \ge t^\star) = 1.$$Then,$$\boxed{\mathrm{FPR}(t^\star) = 0, \quad \mathrm{TPR}(t^\star) = 1,}$$and the ROC curve passes through the upper-left corner $(0,1)$, in addition to the endpoints $(0,0)$ and $(1,1)$.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-perfect-separation')"}[▶]{#proof-arrow-thm-perfect-separation .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-perfect-separation .proof-content}By definition, $\mathrm{FPR}(t^\star) = \mathbb{P}(X_0 \ge t^\star) = 0$ and $\mathrm{TPR}(t^\star) = \mathbb{P}(X_1 \ge t^\star) = 1$. Thus the threshold $t^\star$ yields the operating point $(0,1)$, representing perfect discrimination.:::::::##### ROC curve for an always-wrong classifier:::: {#cor-perfect-anti-separation}# ROC curve for an always-wrong classifierSuppose there exists a threshold $t^\star \in \mathbb{R}$ such that$$\mathbb{P}(X_0 \ge t^\star) = 1, \quad \mathbb{P}(X_1 \ge t^\star) = 0.$$Then,$$\boxed{\mathrm{FPR}(t^\star) = 1, \quad \mathrm{TPR}(t^\star) = 0,}$$and the ROC curve passes through the lower-right corner $(1,0)$, in addition to the endpoints $(0,0)$ and $(1,1)$.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('cor-perfect-anti-separation')"}[▶]{#proof-arrow-cor-perfect-anti-separation .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-cor-perfect-anti-separation .proof-content}By definition, $\mathrm{FPR}(t^\star) = \mathbb{P}(X_0 \ge t^\star) = 1$ and $\mathrm{TPR}(t^\star) = \mathbb{P}(X_1 \ge t^\star) = 0$. Thus the threshold $t^\star$ yields the operating point $(1,0)$, representing perfectly inverted discrimination.:::::::##### ROC curve under random guessing:::: {#thm-random-guessing}# ROC curve under random guessingSuppose $X_0 \stackrel{d}{=} X_1$. Then, for every $t \in \mathbb{R}$,$$\boxed{\mathrm{FPR}(t) = \mathrm{TPR}(t).}$$Consequently, the ROC curve lies on the diagonal line $u=v$ of the unit square.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-random-guessing')"}[▶]{#proof-arrow-thm-random-guessing .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-random-guessing .proof-content}Since $X_0 \stackrel{d}{=} X_1$, $\mathbb{P}(X_0 \ge t) = \mathbb{P}(X_1 \ge t)$ for every $t \in \mathbb{R}$, so $\mathrm{FPR}(t) = \mathrm{TPR}(t)$ for all $t$, and the ROC curve coincides with the diagonal.:::::::#### Monotonicity of ROC curvesA fundamental property of ROC curves, one that distinguishes them from Precision-Recall curves, is their **monotonicity**.:::: {#thm-roc-monotonicity}# Monotonicity of ROC curvesBoth $\mathrm{TPR}(\cdot)$ and $\mathrm{FPR}(\cdot)$ are non-increasing in the threshold: for any $t_1, t_2 \in \mathbb{R}$ with $t_1 < t_2$,$$\begin{align*}\mathrm{TPR}(t_1) & \ge \mathrm{TPR}(t_2),\\\mathrm{FPR}(t_1) & \ge \mathrm{FPR}(t_2).\end{align*}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-roc-monotonicity')"}[▶]{#proof-arrow-thm-roc-monotonicity .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-roc-monotonicity .proof-content}Recall $\mathrm{TPR}(t) = \mathbb{P}(X_1 \ge t)$ and $\mathrm{FPR}(t) = \mathbb{P}(X_0 \ge t)$. Fix $t_1<t_2$: since $X\ge t_2$ implies $X\ge t_1$, the event $\{X\ge t_2\}$ is contained in $\{X\ge t_1\}$ for both $X_0$ and $X_1$. By monotonicity of probability measures,$$\mathbb{P}(X_1 \ge t_2) \le \mathbb{P}(X_1 \ge t_1) \quad\text{and}\quad \mathbb{P}(X_0 \ge t_2) \le \mathbb{P}(X_0 \ge t_1),$$i.e. $\mathrm{TPR}(t_1) \ge \mathrm{TPR}(t_2)$ and $\mathrm{FPR}(t_1) \ge \mathrm{FPR}(t_2)$.:::::::##### Invariance of ROC curves under increasing transformationsA strictly increasing bijection does not introduce new ROC geometry: the ROC curve of $g(X)$ is identical to that of $X$, with only the threshold parametrization transformed via $t\mapsto g^{-1}(t)$.:::: {#thm-roc-bijective-transform}# Invariance of ROC curves under increasing transformationsLet $X$ be a real-valued random variable, with $X\mid Y=0$ and $X\mid Y=1$ atomless, and let $g:\mathbb{R}\to\mathbb{R}$ be a strictly increasing bijection.Writing $$\begin{align*}\mathrm{FPR}_X(t)=\mathbb{P}(X\ge t\mid Y=0),\\ \mathrm{TPR}_X(t)=\mathbb{P}(X\ge t\mid Y=1),\end{align*}$$the operating characteristics of $g(X)$ are$$\boxed{\begin{aligned}\mathrm{FPR}_{g(X)}(t) &= \mathrm{FPR}_X(g^{-1}(t)), \\\mathrm{TPR}_{g(X)}(t) &= \mathrm{TPR}_X(g^{-1}(t)).\end{aligned}}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-roc-bijective-transform')"}[▶]{#proof-arrow-thm-roc-bijective-transform .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-roc-bijective-transform .proof-content}Since $g$ is a strictly increasing bijection, so is $g^{-1}$, and the events $\{g(X)\ge t\}$ and $\{X\ge g^{-1}(t)\}$ coincide: $g(X)\ge t \iff X\ge g^{-1}(t)$. Hence, for $y\in\{0,1\}$,$$\mathbb{P}(g(X)\ge t\mid Y=y) = \mathbb{P}(X\ge g^{-1}(t)\mid Y=y),$$which, specializing to $y=0$ and $y=1$, is exactly the boxed identity. As $t$ ranges over $\mathbb{R}$, so does $g^{-1}(t)$, and $t\mapsto g^{-1}(t)$ is an order-preserving reparametrization of thresholds — so the ROC curve of $g(X)$ is identical to that of $X$, only traced by a relabeled threshold.:::::::#### Symmetries of ROC curvesThe ROC curve exhibits three nontrivial symmetries, generated by reversing the decision rule and/or swapping the class labels. Each acts as an exact affine reflection of the $(\mathrm{FPR},\mathrm{TPR})$ square, mostly because **the ROC curve is completely unaffected by class prevalence**.Each symmetry is described by its action on the operating point $(\mathrm{FPR}(t),\mathrm{TPR}(t))$ of the upper-tail rule $\hat{Y}_t=\mathbb{1}\{X\ge t\}$.No assumption on the laws of $X\mid Y$ is needed here, since $\{X<t\}$ is exactly the complement of $\{X\ge t\}$.##### Threshold reversalKeeping the class labels unchanged and reversing the rule to $\hat{Y}_t^\star=\mathbb{1}\{X<t\}$ sends the operating point as$$\boxed{\binom{\mathrm{FPR}(t)}{\mathrm{TPR}(t)} \longmapsto \binom{1-\mathrm{FPR}(t)}{1-\mathrm{TPR}(t)}.}$$:::: {#thm-roc-threshold-reversal}# Central symmetryThe ROC curve under threshold reversal (class labels fixed) is the central reflection of the original ROC curve about $(0.5,0.5)$.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-roc-threshold-reversal')"}[▶]{#proof-arrow-thm-roc-threshold-reversal .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-roc-threshold-reversal .proof-content}Under the reversed rule $\hat{Y}_t^\star=\mathbb{1}\{X<t\}$,$$\mathbb{P}(X<t\mid Y=0) = 1-\mathrm{FPR}(t), \quad \mathbb{P}(X<t\mid Y=1) = 1-\mathrm{TPR}(t).$$Thus $(\mathrm{FPR}(t),\mathrm{TPR}(t))\mapsto(1-\mathrm{FPR}(t),\,1-\mathrm{TPR}(t))$, a central reflection about $(0.5,0.5)$.:::::::##### Class label swapKeeping the rule unchanged but swapping the labels, $Y^\star=1-Y$, $\hat{Y}_t^\star=\mathbb{1}\{X\ge t\}$, sends the operating point as$$\boxed{\binom{\mathrm{FPR}(t)}{\mathrm{TPR}(t)} \longmapsto \binom{\mathrm{TPR}(t)}{\mathrm{FPR}(t)}.}$$:::: {#thm-roc-class-swap}# Main-diagonal symmetryUnder class-label swapping with the rule unchanged, the ROC curve is the reflection of the original curve about the main diagonal $y=x$.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-roc-class-swap')"}[▶]{#proof-arrow-thm-roc-class-swap .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-roc-class-swap .proof-content}Since $Y^\star=1-Y$, $\{Y^\star=0\}=\{Y=1\}$ and $\{Y^\star=1\}=\{Y=0\}$. Under the unchanged rule,$$\mathbb{P}(X\ge t\mid Y^\star=0) = \mathrm{TPR}(t), \quad \mathbb{P}(X\ge t\mid Y^\star=1) = \mathrm{FPR}(t).$$Thus $(\mathrm{FPR}(t),\mathrm{TPR}(t))\mapsto(\mathrm{TPR}(t),\,\mathrm{FPR}(t))$, a reflection about $y=x$.:::::::##### Class label swap with threshold reversalSwapping the labels **and** reversing the rule, $Y^\star=1-Y$, $\hat{Y}_t^\star=\mathbb{1}\{X<t\}$, sends the operating point as$$\boxed{\begin{pmatrix}\mathrm{FPR}(t)\\ \mathrm{TPR}(t)\end{pmatrix}\longmapsto\begin{pmatrix}1-\mathrm{TPR}(t)\\ 1-\mathrm{FPR}(t)\end{pmatrix}.}$$:::: {#thm-roc-class-inversion}# Anti-diagonal symmetryUnder simultaneous class-label swapping and threshold reversal, the ROC curve is the reflection of the original curve about the anti-diagonal $y=1-x$.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-roc-class-inversion')"}[▶]{#proof-arrow-thm-roc-class-inversion .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-roc-class-inversion .proof-content}As above, $\{Y^\star=0\}=\{Y=1\}$ and $\{Y^\star=1\}=\{Y=0\}$. Under the reversed rule, $\{X<t\}=\{X\ge t\}^c$ gives$$\mathbb{P}(X<t\mid Y^\star=0) = 1-\mathrm{TPR}(t), \quad \mathbb{P}(X<t\mid Y^\star=1) = 1-\mathrm{FPR}(t).$$Thus $(\mathrm{FPR}(t),\mathrm{TPR}(t))\mapsto(1-\mathrm{TPR}(t),\,1-\mathrm{FPR}(t))$, a reflection about $y=1-x$.:::::::##### Summary and visualization::: {style="font-size:90%; overflow-x:auto; margin: 1em 0;"} Transformation | FPR$'$ / TPR$'$ | Geometry | |---|---|---| | Threshold reversal | $1-\mathrm{FPR}$ / $1-\mathrm{TPR}$ | Central symmetry, center $(0.5,0.5)$ | | Label swap | $\mathrm{TPR}$ / $\mathrm{FPR}$ | Main diagonal $y=x$ | | Reversal + swap | $1-\mathrm{TPR}$ / $1-\mathrm{FPR}$ | Anti-diagonal $y=1-x$ |: Transformations of the ROC operating point. Reversal: $X<t \leftrightarrow X\ge t$; label swap: $Y=0\leftrightarrow Y=1$. {#tbl-roc-transformations .striped .hover}:::These three transformations, together with the identity, preserve the curve's overall shape. As noted above, none of them depends on the class prevalence $\pi_1$.You can verify this directly with the interactive below**Interactive ROC curve exploration.** The visualization below illustrates these properties for a Gaussian model $X\mid Y=0\sim\mathcal N(0,1)$, $X\mid Y=1\sim\mathcal N(1,2^2)$. Use the **Threshold** slider to move along the ROC curve and watch the operating point change; the **Prevalence ($\pi_1$)** slider confirms that the ROC curve itself is unaffected by class imbalance, however that slider is moved. The plot also shows the reversed-rule and class-inverted curves described above.::: {#fig-gaussian-symmetric-roc}```{=html}<div id="gaussian-symmetric-roc-interactive" class="gaussian-symmetric-roc-interactive" style="display:flex; gap:10px; align-items:stretch; flex-wrap:wrap; width:100%; max-width:700px; margin:0 auto; overflow-x:auto; font-family:'Inter', -apple-system, 'Segoe UI', Helvetica, Arial, sans-serif;"> <div class="gaussian-symmetric-roc-controls" style="display:flex; flex-direction:column; justify-content:center; min-width:150px; flex:0 0 150px; border:1px solid rgba(0,0,0,0.15); border-radius:8px; padding:8px; font-size:9.5px; background:rgba(0,0,0,0.02);"> </div> <div class="gaussian-symmetric-roc-plot" style="flex:1 1 340px; min-width:340px; display:flex; align-items:center; justify-content:center;"> </div></div><script type="module" src="gaussian_symmetric_roc.js"></script>```**ROC curve under symmetry transformations**:::### ROC curves for common distributionsBelow we summarize the ROC curves for classic location-scale distributions, along with the exponential case.Throughout, we assume that $Z_0, Z_1$ are independent and identically distributed with a common continuous CDF $F$.We define the location-scale variables as:$$\begin{align*}X_0 & = \sigma_0 Z_0 + \mu_0,\\X_1 & = \sigma_1 Z_1 + \mu_1,\end{align*}$$where $\sigma_0 > 0$ and $\sigma_1 > 0$ are the scale parameters, and $\mu_0, \mu_1 \in \mathbb{R}$ are the location parameters. We use the following notation:- $\Delta = (\mu_1 - \mu_0) / \sigma_1$ (standardized separation)- $\rho = \sigma_0 / \sigma_1 > 0$ (scale ratio)- $\Phi(z) = \int_{-\infty}^z \frac{1}{\sqrt{2\pi}} e^{-u^2/2} \, du$ (standard Gaussian CDF)- $\Psi(z) = \frac{1}{2} + \frac{1}{\pi} \arctan(z)$ (Standard Cauchy CDF) and $\Psi^{-1}(u) = \tan\left(\pi\left(u - \frac{1}{2}\right)\right)$.- $[x]_0^1 = \max(0, \min(1, x))$ (clipping function)- $(x)_+= \max(0,x)$::: {style="font-size:90%; margin: 1em 0;"}| **Law of $Z$** | **$F(z)$** | **$\mathrm{TPR}(u)$, $u=\mathrm{FPR}$** ||:---------------|:-----------|:--------------------------------------|| **Gaussian** | $\Phi(z)$ | $\Phi\!\left(\Delta+\rho\Phi^{-1}(u)\right)$ || **Cauchy** | $\Psi(z)$ | $\Psi\left(\Delta + \rho \Psi^{-1}(u)\right)$ || **Logistic** | $\frac{1}{1+e^{-z}}$ | $\frac{e^\Delta u^\rho}{e^\Delta u^\rho+(1-u)^\rho}$ || **Uniform** $(-\tfrac12,\tfrac12)$ | $[z+\frac12]_0^1$ | $[\Delta+\frac{1-\rho}{2}+\rho u]_0^1$ || **Gumbel** | $e^{-e^{-z}}$ | $1-e^{\left[-e^\Delta(-\ln(1-u))^\rho\right]}$ || **Exp.** | $(1-e^{-z})_+$ | $\min(e^\Delta u^\rho,1)$ |: ROC curves for common distributions {#tbl-distributions .striped .hover}:::**Where the table comes from.** All the rows follow from a single identity (@thm-location-scale-roc): for $X_y=\sigma_yZ_y+\mu_y$ with $Z_0,Z_1$ i.i.d. with CDF $F$,$$\mathrm{TPR}\!=\!1\!-\!F\bigl(\rho\,F^{-1}(1\!-\!\mathrm{FPR})\!-\!\Delta\bigr),$$which simplifies to $\mathrm{TPR}=F\bigl(\Delta+\rho\,F^{-1}(\mathrm{FPR})\bigr)$ when $F$ is symmetric (@cor-symmetric-roc). The statements, the proofs, and the row-by-row computations (Gaussian, Cauchy, Logistic, Uniform, Gumbel, Exponential) are collected in [Appendix A](#sec-app-roc).**Visualization**The widget below lets you explore @thm-location-scale-roc interactively for the distribution described in @tbl-distributions.Use the sliders on the left to set $\mu_0,\sigma_0$ (healthy class) and $\mu_1,\sigma_1$ (sick class), and drag the **Threshold** slider to move $t$.- The **middle panel** shows the two population densities plotted vertically: $X_1$ (sick) on the left in blue, $X_0$ (healthy) on the right in red. The horizontal black line marks the current threshold $t$; the darker-shaded region of each density is $\{X<t\}$ (the portion of that class predicted *negative* under the rule $\hat{Y}_t=\mathbb{1}\{X\ge t\}$) while the lighter region above the line is predicted *positive*.- The **right panel** shows the theoretical ROC curve for the current $(\mu_0,\sigma_0,\mu_1,\sigma_1)$, with the dashed diagonal as the random-guessing reference (@thm-random-guessing) and a marker at the operating point $(\mathrm{FPR}(t),\mathrm{TPR}(t))$ corresponding to the threshold you've selected, so you can see exactly which point on the curve a given threshold produces, and watch the curve itself deform as you change the means, standard deviations, or their ratio $\rho=\sigma_0/\sigma_1$.The title above the plot reports the current $\Delta=(\mu_1-\mu_0)/\sigma_1$, $\rho = \sigma_0/\sigma_1$.::: {#fig-gaussian-roc-interactive}```{=html}<div id="gaussian-roc-interactive" class="gaussian-roc-interactive" style="display:flex; gap:10px; align-items:stretch; flex-wrap:nowrap; width:100%; max-width: min(895px, 100%); margin:0 auto; overflow-x:auto; font-family:'Inter', -apple-system, 'Segoe UI', Helvetica, Arial, sans-serif;"> <div class="gaussian-roc-controls" style="display:flex; flex-direction:column; justify-content:center; min-width:125px; flex:0 0 115px; border:1px solid rgba(0,0,0,0.15); border-radius:8px; padding:6px; font-size:9px; background:rgba(0,0,0,0.02);"> </div> <div class="gaussian-roc-plot" style="flex:1 1 auto; min-width:260px; display:flex; align-items:center; justify-content:center;"> </div></div><script type="module" src="gaussian_roc_interactive.js"></script>```**ROC curve for classic distributions**:::**Note**: Visually, the TPR is the ratio of the light blue area over the full blue area on the left, while the FPR is the ratio of the light red area over the full red area (right).**Interpretation**$\Delta=(\mu_1-\mu_0)/\sigma_1$ measures the (scale-standardized) separation between $X_0$ and $X_1$; $\rho=\sigma_0/\sigma_1$ compares their spread. Together they set the difficulty and shape of the ROC curve.* **$\rho=1$:** the usual equal-scale model, $\mathrm{TPR}=F(\Delta+F^{-1}(\mathrm{FPR}))$ for symmetric $F$, e.g., the binormal case $\mathrm{TPR}=\Phi(\Delta+\Phi^{-1}(\mathrm{FPR}))$.* **$\Delta=0,\ \rho=1$:** $X_0\stackrel{d}{=}X_1$, and the ROC curve is the diagonal (random guessing). Note $\Delta=0$ alone is *not* enough when $\rho\ne1$: equal location with unequal scale still separates the classes, via spread rather than location.* **$\rho\ne1$:** the curve need not be symmetric about the anti-diagonal or concave everywhere, if $\rho<1$ (sick population more dispersed), the curve can dip toward the diagonal near one corner before rising, a non-concave S-shape sometimes seen in real assay data.* **$\Delta\to\infty$** (fixed $\rho$): the classes separate perfectly, and the curve approaches the ideal curve through $(0,1)$.Now we illustrate the empirical counterpart, when one samples elements from the distributions but without access to the true densities.::: {#fig-gaussian-roc-interactive-samples}```{=html}<div id="sampling-roc-interactive" class="sampling-roc-interactive" style="display:flex; gap:10px; align-items:stretch; flex-wrap:nowrap; width:100%; max-width: min(895px, 100%)x; margin:0 auto; overflow-x:auto; font-family:'Inter', -apple-system, 'Segoe UI', Helvetica, Arial, sans-serif;"> <div class="sampling-roc-controls" style="display: flex; flex-direction:column; justify-content: center; min-width:130px; flex:0 0 120px; border:1px solid rgba(0,0,0,0.15); border-radius:8px; padding:6px; font-size:9px; background:rgba(0,0,0,0.02);"> </div> <div class="sampling-roc-plot" style="flex:1 1 auto; min-width:260px; display:flex; align-items:center; justify-content:center;"> </div></div><script type="module" src="roc_sampling_explorer.js"></script>```**Empirical counterpart of ROC curve for classic distributions.**:::```{python}#| echo: falsedef roc(thresholds, y_true, score):# Extracted from https://stackoverflow.com/questions/25009284/how-to-plot-roc-curve-in-python fpr = [] # false positive rate tpr = [] # true positive rate# get number of positive and negative examples in the dataset P =sum(y_true) N =len(y_true) - P# iterate through all thresholds and determine fraction of true positives# and false positives found at this thresholdfor thresh in thresholds: FP =0 TP =0for i inrange(len(score)):if score[i] >= thresh:if y_true[i] ==True: TP +=1else: FP +=1 fpr.append(FP /float(N)) tpr.append(TP /float(P))return tpr, fpr```**Note**: the jittering on the x-axis for the scatter plots is not meaningful but only here for visualization.### Area Under the ROC Curve (Cis ranked higher than a randomly chosen negative one, ties counting for one half.Often the AUC is used to measure overall performance: the larger the AUC, the better the assay discriminates between the two classes.Remind that $X_0$ and $X_1$ are independent, distributed as the conditional laws of $X$ given $Y=0$ and $Y=1$, respectively:$$\begin{align*}X_0 & \sim \mathcal{L}(X\mid Y=0)\\X_1 & \sim \mathcal{L}(X\mid Y=1).\end{align*}$$As conditional laws of a real random variable $X$, both $X_0$ and $X_1$ are almost surely finite.Below we use the notation $\overline{\mathbb{R}}= [-\infty, + \infty]$.:::: {#thm-randomized-roc}# Randomized ROC curveFor $s\in[0,1]$, define$$t(s) \triangleq \sup\{t\in\mathbb{R} : \mathbb{P}(X_0\ge t)\ge s\} \in \overline{\mathbb{R}}$$and$$\lambda(s) \triangleq \frac{s - \mathbb{P}(X_0 > t(s))}{\mathbb{P}(X_0 = t(s))} \cdot \mathbb{1}_{\{\mathbb{P}(X_0 = t(s)) > 0\}}$$Define the randomized decision rule$$\phi_s(x) =\begin{cases}1, & x > t(s), \\\lambda(s), & x = t(s), \\0, & x < t(s).\end{cases}$$Then $\lambda(s)\in[0,1]$ is well defined for every $s\in[0,1]$, and$$\mathrm{FPR}(\phi_s) \triangleq \mathbb{E}[\phi_s(X_0)] = s,$$so the (randomized) ROC curve $\mathrm{ROC}(s)\triangleq\mathrm{TPR}(\phi_s)=\mathbb{E}[\phi_s(X_1)]$, $s\in[0,1]$, is well defined for any laws of $X_0$ and $X_1$.Moreover, at every atom $c$ of $X_0$ (i.e., $\mathbb{P}(X_0=c)>0$), $\mathrm{ROC}$ is affine on the interval $\bigl(\mathbb{P}(X_0>c),\ \mathbb{P}(X_0\ge c)\bigr]$, of length $\mathbb{P}(X_0=c)$, with slope $\mathbb{P}(X_1=c)/\mathbb{P}(X_0=c)$.Finally,$$\boxed{\mathrm{AUC} \triangleq \int_0^1\mathrm{ROC}(s)\,ds,}$$and$$\boxed{\mathrm{AUC} = \mathbb{P}(X_1>X_0) + \tfrac{\mathbb{P}(X_1=X_0)}{2}.}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-randomized-roc')"}[▶]{#proof-arrow-thm-randomized-roc .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-randomized-roc .proof-content}Throughout, write $S_0(t)\triangleq\mathbb{P}(X_0\ge t)$ for the survival function of $X_0$, and $S_0(x+)\triangleq\lim_{u\downarrow x}S_0(u)$ for its right limit at any $x\in\mathbb{R}$. As a survival function, $S_0$ is non-increasing, and it is left-continuous, $S_0(x)=\lim_{u\uparrow x}S_0(u)$, since $\{X_0\ge u_n\}\downarrow\{X_0\ge x\}$ along any $u_n\uparrow x$; likewise $\{X_0\ge u_n\}\uparrow\{X_0>x\}$ along any $u_n\downarrow x$ gives $S_0(x+)=\mathbb{P}(X_0>x)$, so$$\mathbb{P}(X_0=x)\triangleq S_0(x)-S_0(x+)=\mathbb{P}(X_0=x)$$is exactly the atomic mass of $X_0$ at $x$. Since $X_0$ is a.s. finite, continuity of measure applied to the nested events $\{X_0\ge t\}$ as $t\to\mp\infty$ also gives$$\lim_{t\to-\infty}S_0(t)=1, \quad \lim_{t\to\infty}S_0(t)=0.$$**Step 1 (existence of $t(s)$).** Call $L\subseteq\mathbb{R}$ a *lower set* if $t\in L,\,u\le t\Rightarrow u\in L$. Fix $s\in(0,1)$ and set $A_s\triangleq\{t:S_0(t)\ge s\}$. Since $S_0$ is non-increasing, $A_s$ is a lower set; since $S_0\to1>s$ as $t\to-\infty$ and $S_0\to0<s$ as $t\to\infty$, $A_s$ is nonempty and bounded above, so$$t(s)\triangleq\sup A_s\ \in\ \mathbb{R}$$is well defined. In fact $A_s=(-\infty,t(s)]$: any $t<t(s)$ is not an upper bound of $A_s$, hence lies in $A_s$ by the lower-set property; and $t(s)\in A_s$ by left-continuity of $S_0$ along $t_n\uparrow t(s)$, $t_n\in A_s$.For the endpoints $s\in\{0,1\}$, the same set $A_s=\{t:S_0(t)\ge s\}$ is used, with the convention $\sup\emptyset=-\infty$: since $S_0\le1$ everywhere, $A_0=\mathbb{R}$ and $t(0)=+\infty$; and $A_1=\{t:S_0(t)=1\}$, which is a (possibly empty) lower set, giving $t(1)\in\mathbb{R}\cup\{-\infty\}$ with $A_1=(-\infty,t(1)]$ by the same argument (or $A_1=\emptyset$, $t(1)=-\infty$, if $S_0(t)<1$ for every $t$).**Step 2 (level sets, and the defining inequality).** For $x\in\mathbb{R}$,$$t(s)=x \quad\Longleftrightarrow\quad S_0(x+) < s \le S_0(x). \tag{$\dagger$}$$*Proof.* By Step 1, $t(s)=x$ if and only if $x\in A_s$ and no $y>x$ lies in $A_s$, i.e.,$$S_0(x)\ge s \quad\text{and}\quad S_0(y)<s\ \ \forall y>x.$$Since $S_0$ is non-increasing, $\sup_{y>x}S_0(y)=S_0(x+)$, so the second condition is equivalent to $s>S_0(x+)$; this proves $(\dagger)$.Taking $x=t(s)$ in $(\dagger)$ yields both the defining inequality and the atom size at the threshold:$$\begin{cases}S_0(t(s)+) \ \le\ s\ \le\ S_0(t(s)), \\S_0(t(s))-S_0(t(s)+) = \mathbb{P}(X_0=t(s)). \tag{$\ast$}\end{cases}$$**Step 3 ($\phi_s$ attains $\mathrm{FPR}(\phi_s)=s$ exactly).** By $(\ast)$,$$0\ \le\ s-S_0(t(s)+)\ \le\ \mathbb{P}(X_0=t(s)),$$so whenever $\mathbb{P}(X_0=t(s))>0$, $\lambda(s)=\dfrac{s-S_0(t(s)+)}{P(X_0=t(s))}\in[0,1]$ . If instead $\mathbb{P}(X_0=t(s))=0$, the two bounds coincide and force $s=S_0(t(s)+)$; the definition sets $\lambda(s)\triangleq 0$ in this case, and its value has no impact since it multiplies the null event $\{X_0=t(s)\}$. In either case,$$\mathrm{FPR}(\phi_s) = \mathbb{E}[\phi_s(X_0)] = S_0(t(s)+) + \lambda(s)\,\mathbb{P}(X_0=t(s)) = s.$$**Step 4 (affine property of ROC).** Fix $c$ with $\mathbb{P}(X_0=c)>0$, and let $s\in(S_0(c+),S_0(c)]$. By $(\dagger)$, $t(s)=c$ throughout this interval, so substituting $t(s)=c$ into the definition of $\lambda$, legitimate since $\mathbb{P}(X_0=t(s))=\mathbb{P}(X_0=c)>0$ there, gives$$\lambda(s) = \frac{s-S_0(c+)}{P(X_0=c)},$$a linear, increasing function of $s$, running from $0$ at $s=S_0(c+)$ to $1$ at $s=S_0(c)$. Hence$\mathrm{ROC}(s) = \mathbb{E}[\phi_s(X_1)] = \mathbb{P}(X_1>c) + \lambda(s)\,\mathbb{P}(X_1=c)$is affine in $s$ on $(S_0(c+),S_0(c)]$, an interval of length $\mathbb{P}(X_0=c)$, with slope $\mathbb{P}(X_1=c)/\mathbb{P}(X_0=c)$.**Step 5 (AUC).** Since $t(\cdot)$ is non-increasing (its sublevel sets $A_s$ are nested in $s$, by Step 1) and $\lambda(\cdot)$ is Borel where defined, $(s,x)\mapsto\phi_s(x)$ is jointly measurable, $[0,1]$-valued, so Tonelli's theorem gives$$\mathrm{AUC} = \int_0^1 \mathbb{E}[\phi_s(X_1)]\,ds = \mathbb{E}\!\left[\int_0^1 \phi_s(X_1)\,ds\right].$$Fix $x\in\mathbb{R}$. By $(\dagger)$, up to the single (Lebesgue-null) point $S_0(x+)$, the interval $[0,1]$ splits into three sets, according to the position of $t(s)$ relative to $x$:$$\begin{align*}\{s:t(s)>x\} &= [0,S_0(x+)], & \operatorname{Leb}\bigl(\{s:t(s)>x\}\bigr) &= \mathbb{P}(X_0>x), \\\{s:t(s)=x\} &= (S_0(x+),S_0(x)], & \operatorname{Leb}\bigl(\{s:t(s)=x\}\bigr) &= \mathbb{P}(X_0=x), \\\{s:t(s)<x\} &= (S_0(x),1], & \operatorname{Leb}\bigl(\{s:t(s)<x\}\bigr) &= \mathbb{P}(X_0<x).\end{align*}$$On each of these three sets, $\phi_s(x)=\mathbb{1}\{x>t(s)\}+\lambda(s)\mathbb{1}\{x=t(s)\}$ takes a simple form:$$\phi_s(x) =\begin{cases}0, & t(s)>x,\\\lambda(s), & t(s)=x,\\1, & t(s)<x.\end{cases}$$Hence $\int_0^1\phi_s(x)\,ds$ splits into three integrals, one over each set:$$\int_0^1\phi_s(x)\,ds= \underbrace{\int_{\{t(s)>x\}} 0\,ds}_{=\,0}\ +\ \underbrace{\int_{\{t(s)=x\}} \lambda(s)\,ds}_{=:I(x)}\ +\ \underbrace{\int_{\{t(s)<x\}} 1\,ds}_{=\,\mathbb{P}(X_0<x)}.$$It remains to compute $I(x)=\int_{(S_0(x+),S_0(x)]} \lambda(s)\,ds$, by splitting on whether $x$ is an atom of $X_0$:- If $\mathbb{P}(X_0=x)=0$: the domain of integration is Lebesgue-null, so $I(x)=0=\tfrac12\,\mathbb{P}(X_0=x)$.- If $\mathbb{P}(X_0=x)>0$: by Step 4 (with $c=x$), $\lambda(s)$ runs linearly from $0$ to $1$ as $s$ ranges over $(S_0(x+),S_0(x)]$. The average of a function running linearly between $0$ and $1$ over an interval is $\tfrac12$ (the midpoint of its range), so $I(x)=\mathbb{P}(X_0=x)\cdot\tfrac12$.Either way, $I(x)=\tfrac12\,\mathbb{P}(X_0=x)$, and therefore$$\int_0^1 \phi_s(x)\,ds = \mathbb{P}(X_0<x) + \tfrac12\,\mathbb{P}(X_0=x) = \mathbb{P}(X_0<x) + \tfrac12\,\mathbb{P}(X_0=x).$$Substituting $x=X_1$, taking expectations, and using the tower property,$$\mathrm{AUC} = \mathbb{P}(X_0<X_1) + \tfrac12\mathbb{P}(X_0=X_1) = \mathbb{P}(X_1>X_0) + \tfrac12\mathbb{P}(X_1=X_0).$$:::::::For continuous random variables (with no point masses), the AUC reduces to the probability that $X_1 > X_0$.:::: {#cor-continuous-auc}# AUC for continuous distributionsLet $X_0 \sim \mathcal{L}(X \mid Y=0)$ and $X_1 \sim \mathcal{L}(X \mid Y=1)$ be independent continuous random variables (no point masses).Then, the AUC of the ROC curve simplifies to:$$\boxed{\mathrm{AUC} = \mathbb{P}(X_1 > X_0).}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('cor-continuous-auc')"}[▶]{#proof-arrow-cor-continuous-auc .proof-arrow} **Proof**:::::: {#proof-content-cor-continuous-auc .proof-content}Since $X_0$ and $X_1$ are continuous, $\mathbb{P}(X_1 = X_0) = 0$. By the randomized ROC theorem (@thm-randomized-roc), the AUC is:$$\mathrm{AUC} = \mathbb{P}(X_1 > X_0) + \tfrac12 \mathbb{P}(X_1 = X_0) = \mathbb{P}(X_1 > X_0).$$:::::::Replacing $X_0,X_1$ by two independent samples and $\mathbb{P}(X_1>X_0)$ by the corresponding sample proportion gives exactly the classical **Mann-Whitney $U$ statistic** from nonparametric statistics, providing the empirical, plug-in counterpart to @cor-continuous-auc.:::: {#thm-empirical-auc-mann-whitney}# Empirical AUC and Mann-Whitney U StatisticLet $X_{1,1}, \dots, X_{1,n_1}$ be an i.i.d. sample from $X_1 \sim \mathcal{L}(X \mid Y=1)$ and $X_{0,1}, \dots, X_{0,n_0}$ be an i.i.d. sample from $X_0 \sim \mathcal{L}(X \mid Y=0)$, with both samples independent of each other. Assume $X_1$ and $X_0$ are continuous random variables (no point masses).Let $R_1$ be the sum of the ranks of the $X_{1,i}$ values in the combined sample of size $n_1 + n_0$ (ranks assigned in ascending order, so the smallest value has rank $1$). The **Mann-Whitney U statistic** for the positive group is defined as:$$\begin{align*}U_1 & = R_1 - \frac{n_1 (n_1 + 1)}{2}\\ & = \sum_{i=1}^{n_1}\sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}).\end{align*}$$Then, the **empirical AUC** of the ROC curve is equivalent to the normalized Mann-Whitney U statistic:$$\boxed{\widehat{\mathrm{AUC}} = \frac{1}{n_1 n_0}\sum_{i=1}^{n_1}\sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}).}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-empirical-auc-mann-whitney')"}[▶]{#proof-arrow-thm-empirical-auc-mann-whitney .proof-arrow} **Proof**:::::: {#proof-content-thm-empirical-auc-mann-whitney .proof-content}1. **Rank Sum Interpretation**: The term $R_1$ in $U_1$ is the sum of the ranks of the $X_{1,i}$ values in the combined sample of size $n_1 + n_0$. For continuous distributions, all $X_{1,i}$ and $X_{0,j}$ are distinct almost surely, so the rank of each $X_{1,i}$ counts: the $X_{0,j}$ values it exceeds, the $X_{1,i'}$ values ($i' \neq i$) it exceeds, and $1$ for its own position. Introducing $$ \begin{align*} S_i & \triangleq \sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}),\\ C_i & \triangleq \sum_{i' \neq i} \mathbb{1}(X_{1,i} > X_{1,i'}), \end{align*} $$ this reads $$ R_1 = \sum_{i=1}^{n_1} \left( S_i + C_i + 1 \right). $$2. **Simplification of $R_1$**: For each unordered pair $\{i,i'\}\subseteq\{1,\dots,n_1\}$, exactly one of the two indices has the larger value, so $\sum_i C_i$ counts each of the $\binom{n_1}{2}$ pairs exactly once: $$ \sum_{i=1}^{n_1} C_i = \binom{n_1}{2} = \frac{n_1(n_1-1)}{2}. $$ Writing $$ S \triangleq \sum_{i=1}^{n_1}\sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}) $$ for the number of cross-sample pairs where the positive-group value is larger, this gives $$ R_1 = S + \frac{n_1(n_1-1)}{2} + n_1 = S + \frac{n_1(n_1+1)}{2}. $$3. **Link to $U_1$**: Substituting this expression for $R_1$ into the definition of $U_1$, $$ U_1 = \left(S + \frac{n_1(n_1+1)}{2}\right) - \frac{n_1(n_1+1)}{2} = S, $$ so $U_1$ is exactly the count of cross-sample pairs with $X_{1,i}>X_{0,j}$: $$ U_1 = \sum_{i=1}^{n_1} \sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}). $$4. **Empirical AUC**: Recall that $\widehat{\mathrm{AUC}}$ is defined as the fraction of positive/negative pairs correctly ordered by the score: $$ \widehat{\mathrm{AUC}} = \frac{1}{n_1 n_0} \sum_{i=1}^{n_1} \sum_{j=1}^{n_0} \mathbb{1}(X_{1,i} > X_{0,j}) = \frac{S}{n_1n_0}. $$ By Step 3, $S=U_1$, so $$ \widehat{\mathrm{AUC}} = \frac{U_1}{n_1 n_0}. $$:::::::We now determine how the three symmetries of the previous section act on the AUC.:::: {#thm-auc-class-inversion}# AUC under threshold reversal and class inversionAssume $X\mid Y=0$ and $X\mid Y=1$ are atomless and almost surely finite. Write $\mathrm{AUC}^\star$ for the AUC of the transformed problem in each case below. Then:- under threshold reversal alone, $\boxed{\mathrm{AUC}^\star = 1-\mathrm{AUC}}$;- under class label swap alone, $\boxed{\mathrm{AUC}^\star = 1-\mathrm{AUC}}$;- under both together (the anti-diagonal transform), $\boxed{\mathrm{AUC}^\star = \mathrm{AUC}}$, leaving the AUC unchanged.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-auc-class-inversion')"}[▶]{#proof-arrow-thm-auc-class-inversion .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-auc-class-inversion .proof-content}Write $u\triangleq\mathrm{FPR}(t)$, $f(u)\triangleq\mathrm{TPR}(t)$, so $\mathrm{AUC}=\int_0^1 f(u)\,du$, with $f$ non-decreasing (@thm-roc-monotonicity), $f(0)=0$, $f(1)=1$ (@lem-roc-endpoints). Let $R\triangleq\{(x,y):0\le x\le1,\,0\le y\le f(x)\}$, of area $\mathrm{AUC}$.**Threshold reversal.** The transformed curve is $g(x')\triangleq 1-f(1-x')$. Reflecting $R$ through $(0.5,0.5)$ sends $(x,y)\mapsto(1-x,1-y)$, and $0\le y\le f(x)$ becomes $1-f(1-x')\le y'\le1$, the region **above** $g$, of area $1-\mathrm{AUC}^\star$. Since central reflection preserves area, $1-\mathrm{AUC}^\star=\mathrm{AUC}$, *i.e.*, $\mathrm{AUC}^\star=1-\mathrm{AUC}$.**Class label swap.** The transformed curve is $g\triangleq f^{-1}$ (generalized inverse). Reflecting $R$ across $y=x$ sends $(x,y)\mapsto(y,x)$, and $0\le x\le f(y)$ becomes, by monotonicity of $f$, $y\ge f^{-1}(x)=g(x)$, the same region above $g$, of area $1-\mathrm{AUC}^\star$. Reflection across a line preserves area, so once more $\mathrm{AUC}^\star=1-\mathrm{AUC}$.**Both together.** The two transforms commute at the level of the operating point, so applying one after the other composes the two identities just derived: $\mathrm{AUC}\mapsto1-\mathrm{AUC}\mapsto1-(1-\mathrm{AUC})=\mathrm{AUC}$.:::::::**Interpretation.** Only the composite (anti-diagonal) transform preserves AUC exactly; the two individual transforms each send $A\mapsto1-A$. This is intuitive: reversing the rule alone, or swapping the labels alone, each turns a good classifier into a bad one for the *new* problem (a classifier that correctly separates $X_0$ from $X_1$ under the original labeling is simply mislabeling everything once only one of the two conventions is flipped). On the other hand, flipping *both* conventions together restores consistency, since relabeling "positive" to mean "low score" and simultaneously testing for "low score" recovers a description of the exact same underlying discrimination ability. The fixed point of $A\mapsto1-A$ at $A=\tfrac12$ (random guessing) is the one case where all three transforms leave AUC unchanged, since $1-\tfrac12=\tfrac12$.You can verify this directly with the interactive above, see @fig-gaussian-symmetric-roc: the "ROC curve" and "Class swap + threshold rev." curves in the plot share the same AUC, while "Threshold reversal" and "Class label swap" alone each trace out the complementary area.## The precision–recall (PR) curveThe **precision–recall (PR)** curve has a different origin from the ROC curve: it comes from *information retrieval*, not from signal detection. In the 1950s, the first automated literature-searching systems raised a practical question: how should one measure the quality of a search? @kent1955machine proposed to count the share of the relevant documents that are retrieved (what we now call *recall*) and the share of the retrieved documents that are relevant (a first version of *precision*, then called "pertinency"). Medicine had its own names for the same quantities: recall is the *sensitivity*, and precision is the positive predictive value.:::: {#def-pr}# Precision–recall (PR) curveFor a threshold $t\in\mathbb{R}$, the **precision–recall (PR) operating point** is the pair of recall and precision,$$\begin{align*}\mathrm{Recall}(t)& = \mathbb{P}(X_1\geq t) \\\mathrm{Precision}(t)& = \mathbb{P}(Y=1\mid \hat{Y}_t=1),\end{align*}$$where $\hat{Y}_t=\mathbb{1}\{X\geq t\}$ denotes the threshold-based classifier.The **PR curve** is the parametric curve traced by this operating point as $t$ varies over $\mathbb{R}$:$$\left\{\bigl(\mathrm{Recall}(t),\mathrm{Precision}(t)\bigr):t\in\mathbb{R}\right\}.$$::::Unlike recall (=TPR) and the FPR, which are conditional on the true class, precision depends on the prevalence $\pi_1$ of the positive class in the population being evaluated.Consequently, the PR curve can change when prevalence changes, even if the class-conditional score distributions remain the same.The empirical recall $\widehat{\mathrm{Recall}}(t)=\mathrm{TP}(t)/n_1$ and precision $\widehat{\mathrm{Precision}}(t)=\mathrm{TP}(t)/\hat{\mathrm{P}}_t$ are the plug-in estimators of the naming section ($\widehat{\mathrm{Recall}}=\widehat{\mathrm{TPR}}$). If no observation is predicted positive, then $\hat{\mathrm{P}}_t=0$ and the empirical precision is undefined: implementations of PR curves adopt a convention for this endpoint, which should be considered when comparing empirical curves.The following theorem makes explicit how prevalence enters the precision formula.:::: {#thm-precision-formula}# Precision in terms of TPR, FPR, and prevalenceLet $\pi_1=\mathbb{P}(Y=1)$ and $\pi_0=\mathbb{P}(Y=0)$. For every threshold $t$ such that $\mathbb{P}(X\ge t)>0$,$$\boxed{\mathrm{Precision}(t)=\tfrac{\pi_1\,\mathrm{TPR}(t)}{\pi_1\,\mathrm{TPR}(t)+\pi_0\,\mathrm{FPR}(t)}.}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-precision-formula')"}[▶]{#proof-arrow-thm-precision-formula .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-precision-formula .proof-content}By the law of total probability,$$\begin{aligned}\mathbb{P}(X\ge t)&=\mathbb{P}(X\ge t\mid Y=1)\pi_1\\&\quad+\mathbb{P}(X\ge t\mid Y=0)\pi_0\\&=\pi_1\,\mathrm{TPR}(t)+\pi_0\,\mathrm{FPR}(t).\end{aligned}$$Furthermore,$$\begin{aligned}\mathbb{P}(Y=1,\ X\ge t)&=\mathbb{P}(X\ge t\mid Y=1)\pi_1\\&=\pi_1\,\mathrm{TPR}(t).\end{aligned}$$By the definition of conditional probability,$$\mathrm{Precision}(t)=\frac{\mathbb{P}(Y=1,\ X\ge t)}{\mathbb{P}(X\ge t)}.$$Substituting the expressions above yields the claimed formula.:::::::For fixed class-conditional score distributions, $\mathrm{TPR}(t)$ and $\mathrm{FPR}(t)$ do not change with prevalence. The ROC curve therefore remains unchanged, whereas **the PR curve generally changes through $\pi_1$ and $\pi_0$** in the theorem above. In particular, precision at a fixed threshold typically decreases as positive cases become less common.In our COVID-19 example, the 976 pre-pandemic negative samples and 536 confirmed positive samples reflect the benchmark's sampling design, not necessarily the prevalence of infection in a clinical population. Consequently, its empirical PR curve and average precision need not represent predictive performance in a target population with a different class composition. The initial dataset description gives the benchmark's positive fraction; it should not be interpreted as clinical prevalence.The Python code to compute this curve is provided below.```{python}def precision_recall(thresholds, y_true, score): precision = [] recall = [] P =sum(y_true)for thresh in thresholds: FP =0 TP =0for i inrange(len(score)):if score[i] >= thresh:if y_true[i] ==True: TP +=1else: FP +=1 recall.append(TP /float(P))# Convention: precision is set to 1 when nothing is predicted positive# (TP + FP = 0), as in scikit-learn at the recall = 0 endpoint. precision.append(TP /float(TP + FP) if (TP + FP) >0else1.0)return precision, recall```### Properties#### EndpointsThroughout this section, assume $\pi_1\in(0,1)$, $\pi_0=1-\pi_1$, and that $X_0,X_1$ are almost surely finite. Recall and precision are as defined above; precision is defined only when $\mathbb{P}(X\ge t)>0$.:::: {#lem-pr-endpoint}# Universal endpoint of the PR curveThere exists $t_0\in\mathbb{R}$ such that the denominator of $\mathrm{Precision}(t)$ is positive for all $t\le t_0$, and$$\bigl(\mathrm{Recall}(t),\mathrm{Precision}(t)\bigr)\mathrel{\underset{t\to-\infty}{\longrightarrow}} (1,\pi_1).$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('lem-pr-endpoint')"}[▶]{#proof-arrow-lem-pr-endpoint .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-lem-pr-endpoint .proof-content}Since $X_0$ and $X_1$ are almost surely finite, the events $\{X_y\ge t\}$ increase to $\{X_y>-\infty\}$, which has probability $1$, as $t\downarrow-\infty$. By continuity of probability measures,$$\lim_{t\to-\infty}\mathbb{P}(X_y\ge t)=1,\quad y\in\{0,1\}.$$Hence $\mathrm{Recall}(t)\to1$, and the denominator of $\mathrm{Precision}(t)$ tends to $\pi_1+\pi_0=1$. In particular it is positive for all $t$ sufficiently negative, so $\mathrm{Precision}(t)$ is defined there. Since the limit of the denominator is $1\neq0$, the quotient rule for limits gives$$\mathrm{Precision}(t)\longrightarrow\frac{\pi_1\cdot1}{\pi_1\cdot1+\pi_0\cdot1}=\pi_1.$$:::::::**Remark (the other end, $\mathrm{Recall}\to0$).** The limit as $t\to+\infty$ is *not* universal: numerator and denominator of $\mathrm{Precision}(t)$ both tend to $0$, and$$\mathrm{Precision}(t)=\Bigl(1+\frac{\pi_0}{\pi_1}\,\frac{\mathbb{P}(X_0\ge t)}{\mathbb{P}(X_1\ge t)}\Bigr)^{-1}$$is governed by the tail ratio $\mathbb{P}(X_0\ge t)/\mathbb{P}(X_1\ge t)$. For instance:- $X_y\sim\mathcal N(\mu_y,\sigma^2)$ with $\mu_1>\mu_0$: the ratio tends to $0$, and the curve ends at $(0,1)$.- $X_0\stackrel{d}{=}X_1$: the ratio is $\equiv1$, and the curve ends at $(0,\pi_1)$.- $X_y\sim\mathcal N(\mu_y,\sigma^2)$ with $\mu_1<\mu_0$: the ratio tends to $+\infty$, and the curve ends at $(0,0)$.### Extreme cases#### Perfect classifier:::: {#thm-perfect-pr-separation}# PR curve for a perfect classifierSuppose there exists a threshold $t^\star \in \mathbb{R}$ such that$$\mathbb{P}(X_0 \ge t^\star) = 0, \quad \mathbb{P}(X_1 \ge t^\star) = 1.$$Then,$$\boxed{\mathrm{Recall}(t^\star) = 1, \quad \mathrm{Precision}(t^\star) = 1,}$$For every $t\ge t^\star$ with $\mathrm{Recall}(t)>0$, precision is $1$. Thus the curve traces the horizontal segment at precision $1$ from $(1,1)$ toward $(0,1)$; at a threshold with no predicted positives, population precision is undefined, although PR-curve implementations commonly use $1$ for this endpoint. As $t\to-\infty$, the curve also approaches the universal endpoint $(1,\pi_1)$.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-perfect-pr-separation')"}[▶]{#proof-arrow-thm-perfect-pr-separation .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-perfect-pr-separation .proof-content}**The point at $t^\star$.** By definition,$$\begin{align*}\mathrm{Recall}(t^\star)& = \mathbb{P}(X_1 \ge t^\star) = 1, \\\mathrm{Precision}(t^\star)& = \frac{\pi_1 \cdot 1}{\pi_1 \cdot 1 + \pi_0 \cdot 0} = 1.\end{align*}$$**The flat segment.** By @thm-roc-monotonicity, $\mathrm{FPR}(t)=\mathbb{P}(X_0\ge t)$ is non-increasing in $t$. Since $\mathrm{FPR}(t^\star)=0$ and $\mathrm{FPR}\ge0$ always, monotonicity gives $\mathrm{FPR}(t)\le\mathrm{FPR}(t^\star)=0$ for every $t\ge t^\star$, so$$\mathrm{FPR}(t) = 0 \quad\text{for all } t\ge t^\star.$$Consequently, for any $t\ge t^\star$ with $\mathrm{TPR}(t)>0$,$$\mathrm{Precision}(t) = \frac{\pi_1\,\mathrm{TPR}(t)}{\pi_1\,\mathrm{TPR}(t) + \pi_0\cdot0} = 1.$$If $\mathrm{TPR}(t)=0$ as well, then no positives are predicted and population precision is undefined. Thus the population PR curve has precision $1$ on the nonempty predicted-positive part of this threshold range; empirical implementations may separately assign precision $1$ at the no-predictions endpoint. **The endpoints.** As $t$ ranges over $[t^\star,\infty)$, recall decreases from $1$ to $0$, and precision remains $1$ whenever predicted positives remain. The endpoint at recall $0$ is represented as $(0,1)$ under the usual PR-plotting convention; population precision is undefined if no positives are predicted. As $t\to-\infty$, $\mathrm{TPR}(t)\to1$ and $\mathrm{FPR}(t)\to1$ (@lem-roc-endpoints), so precision tends to $\pi_1$: the universal endpoint $(1,\pi_1)$ (see @thm-precision-formula).:::::::#### Always-wrong classifier@thm-always-wrong-pr pins down the PR curve only for $t\ge t^\star$ (a single point, via monotonicity of $\mathrm{TPR}$). The same hypothesis, applied to $\mathrm{FPR}$ instead, determines the remaining branch $t\le t^\star$ as well.:::: {#thm-always-wrong-pr}# PR curve for an always-wrong classifierSuppose there exists $t^\dagger \in \mathbb{R}$ such that$$\mathbb{P}(X_0 \ge t^\dagger) = 1, \quad \mathbb{P}(X_1 \ge t^\dagger) = 0$$Then the operating points satisfy the following:- for $t\ge t^\dagger$: $\mathrm{Recall}(t)\equiv0$; precision is $0$ when $\mathrm{FPR}(t)>0$ and is undefined when no positives are predicted;- for $t\le t^\dagger$: $\mathrm{FPR}(t)\equiv1$, and with $r\triangleq \mathrm{Recall}(t)$, every attainable recall value satisfies$$\boxed{\mathrm{Precision}(r) = \frac{\pi_1\,r}{\pi_1\,r + \pi_0}.}$$If the positive-class score distribution is continuous, these points trace the full curve for $r\in[0,1]$.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-always-wrong-pr')"}[▶]{#proof-arrow-thm-always-wrong-pr .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-always-wrong-pr .proof-content}**Branch $t\ge t^\dagger$.** By the assumption, $\mathrm{TPR}(t^\dagger)=0$; monotonicity (@thm-roc-monotonicity) gives $\mathrm{TPR}(t)=0$ for every $t\ge t^\dagger$. Hence recall is $0$ throughout this range. Precision is $0$ when $\mathrm{FPR}(t)>0$ and is undefined if $\mathrm{FPR}(t)=0$, because then no positives are predicted. Under the common plotting convention that assigns precision $1$ when no positives are predicted, the limiting endpoint is represented as $(0,1)$; the population definition itself leaves precision undefined there.**Branch $t\le t^\dagger$.** Since $\mathrm{FPR}(t^\dagger)=1$ and $\mathrm{FPR}$ is non-increasing, $\mathrm{FPR}(t)\ge1$ for $t\le t^\dagger$, forcing $\mathrm{FPR}(t)\equiv1$ there. Substituting into @thm-precision-formula,$$\begin{align*}\mathrm{Precision}(t) & = \frac{\pi_1\,\mathrm{TPR}(t)}{\pi_1\mathrm{TPR}(t)+\pi_0}\\& = \frac{\pi_1\,r}{\pi_1\,r+\pi_0}, \quad r\triangleq \mathrm{Recall}(t).\end{align*}$$As $t$ decreases from $t^\dagger$ to $-\infty$, $r=\mathrm{TPR}(t)$ increases from $0$ to $1$ (@lem-roc-endpoints), sweeping the boxed curve up to $(1,\pi_1)$.:::::::#### Random guessing:::: {#thm-random-pr}# PR curve under random guessingSuppose $X_0 \stackrel{d}{=} X_1$. Write $p(t)\triangleq \mathbb{P}(X_0\ge t)=\mathbb{P}(X_1\ge t)$. Then $\mathrm{Precision}(t)$ is defined if and only if $p(t)>0$, and in that case$$\boxed{\mathrm{Recall}(t)=p(t), \quad \mathrm{Precision}(t) = \pi_1.}$$Consequently the PR curve $\{(\mathrm{Recall}(t),\mathrm{Precision}(t)) : p(t)>0\}$ is contained in the horizontal line $\mathrm{Precision}=\pi_1$.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-random-pr')"}[▶]{#proof-arrow-thm-random-pr .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-random-pr .proof-content}By definition, $\mathrm{Recall}(t)=\mathbb{P}(X_1\ge t)$ and$$\begin{align*}\mathrm{Precision}(t)& =\mathbb{P}(Y=1\mid\hat{Y}_t=1)\\& =\frac{\pi_1\,\mathbb{P}(X_1\ge t)}{\pi_1\,\mathbb{P}(X_1\ge t)+\pi_0\,\mathbb{P}(X_0\ge t)},\end{align*}$$which is defined if and only if the denominator $\mathbb{P}(\hat{Y}_t=1)$ is positive. Since $X_0\stackrel d=X_1$, both tail probabilities equal $p(t)$, so $\mathrm{Recall}(t)=p(t)$ and the denominator equals $(\pi_1+\pi_0)\,p(t)=p(t)$. Hence $\mathrm{Precision}(t)$ is defined if and only if $p(t)>0$, and then$$\mathrm{Precision}(t)=\frac{\pi_1\,p(t)}{p(t)}=\pi_1.$$:::::::**Interpretation.** The three cases show an asymmetry between the two ends of the PR curve.- **$\mathrm{Recall}=1$ is the universal endpoint.** It is reached as $t\to-\infty$, where the classifier labels *everything* positive. Precision is then simply the base rate, $\mathrm{Precision}=\pi_1$, whatever the score distributions are (@lem-roc-endpoints + @thm-precision-formula, no extra hypothesis). The perfect, random and always-wrong classifiers all pass through $(1,\pi_1)$.- **$\mathrm{Recall}=0$ is model-dependent.** It is reached as $t\to+\infty$, where the classifier labels almost nothing positive, and $\mathrm{Precision}$ measures the purity of the *extreme upper tail* of the scores. That purity is not determined by $\pi_1$ or by the ROC endpoints alone, but by the tail ratio $\mathbb{P}(X_0\ge t)/\mathbb{P}(X_1\ge t)$. The limit can take any value in $[0,1]$: $1$ for the perfect classifier (the top scores are all positives), $\pi_1$ for random guessing (the top scores carry no information), and $0$ for the always-wrong classifier (the top scores are all negatives).This is why the earlier Gaussian and exponential derivations needed tail-rate arguments (Mills' ratio; a power-law comparison) to determine the $\mathrm{Recall}=0$ end. Unlike the $\mathrm{Recall}=1$ endpoint, it is never given for free.### Symmetries of PR curvesUnlike in ROC space, none of the three transformations (threshold reversal, class label swap, and their composition) has a simple geometric description in PR space: there is no reflection or rotation of the original PR curve that produces the transformed curves. The map $(\mathrm{FPR},\mathrm{TPR})\mapsto(\mathrm{Recall},\mathrm{Precision})$ is nonlinear and depends on the prevalence $\pi_1=\mathbb{P}(Y=1)$, and each transformation also changes the role of the positive class.Each transformed curve must therefore be recomputed from the definitions of precision and recall. Consequently, the transformed curves generally have a different shape from the original one, and, unlike the ROC curve, they all depend on $\pi_1$. What remains simple is the endpoint at $\mathrm{Recall}=1$: it is $(1,\pi_1)$ for the original curve and the threshold reversal, and $(1,1-\pi_1)$ for the two curves with swapped labels.You can verify this directly with the interactive below.**Interactive PR curve exploration.** The visualization below illustrates these properties for the same Gaussian model $X\mid Y=0\sim\mathcal N(0,1)$, $X\mid Y=1\sim\mathcal N(1,2^2)$ as before. Use the **Threshold ($t$)** slider to move along the PR curve and watch the operating point change, together with the corresponding points on the transformed curves. The **Prevalence ($\pi_1$)** slider shows that, unlike the ROC curve, the PR curve and its transformations are reshaped by the class balance, and the endpoints at $\mathrm{Recall}=1$ move with it. The plot also shows the threshold-reversed and class-swapped curves described above, as well as the extreme cases (perfect, random and always-wrong classifiers). Click an entry in the legend to show or hide the corresponding curve.::: {#fig-gaussian-symmetric-pr}```{=html}<div id="gaussian-symmetric-pr-interactive" class="gaussian-symmetric-pr-interactive" style="display:flex; gap:10px; align-items:stretch; flex-wrap:wrap; width:100%; max-width:700px; margin:0 auto; overflow-x:auto; overflow-x:auto; font-family:'Inter', -apple-system, 'Segoe UI', Helvetica, Arial, sans-serif;"> <div class="gaussian-symmetric-pr-controls" style="display:flex; flex-direction:column; justify-content:center; min-width:150px; flex:0 0 150px; border:1px solid rgba(0,0,0,0.15); border-radius:8px; padding:8px; font-size:9.5px; background:rgba(0,0,0,0.02);"> </div> <div class="gaussian-symmetric-pr-plot" style="flex:1 1 340px; min-width:340px; display:flex; align-items:center; justify-content:center;"> </div></div><script type="module" src="gaussian_symmetric_pr.js"></script>```**PR curve under symmetry transformations**:::### PR curves for common distributionsWe now summarize the PR curves of the same location-scale families as in the ROC case (@tbl-distributions), with the same notation: $X_y=\sigma_yZ_y+\mu_y$ with $Z_0,Z_1$ i.i.d. with CDF $F$, $\Delta=(\mu_1-\mu_0)/\sigma_1$, $\rho=\sigma_0/\sigma_1$, $\Phi$, $[x]_0^1$ and $(x)_+$. We additionally assume that $F$ is continuous and strictly increasing on $\{z:0<F(z)<1\}$, which holds for every law in the table below. Unlike the ROC curve, the PR curve also depends on the prevalence $\pi_1$.**Parametrization by recall.** Since $\mathrm{Recall}(t)=\mathrm{TPR}(t)=1-F\bigl(\tfrac{t-\mu_1}{\sigma_1}\bigr)$, the assumption on $F$ makes $\mathrm{Recall}$ a continuous, strictly decreasing bijection from $\{t:0<\mathrm{Recall}(t)<1\}$ onto $(0,1)$. Hence, for every recall level $r\in(0,1)$, there is a **unique** threshold $t_r$ with $\mathrm{Recall}(t_r)=r$, and the PR curve can be parametrized by $r$. We define$$\begin{align*}G(r)& \triangleq \mathrm{FPR}(t_r)\\& =\mathbb{P}(X_0\ge t_r).\end{align*}$$Thus $G$ expresses the ROC curve in terms of recall: it gives the false-positive rate required to attain a given true-positive rate. Like the ROC curve, $G$ depends only on the class-conditional distributions, not on the prevalence $\pi_1$.The endpoints $r=0$ and $r=1$ are excluded on purpose, because uniqueness may fail there. For instance, for the Uniform and Exponential laws, $\mathrm{Recall}(t)=1$ for *all* thresholds $t$ below the support, so there is no single $t_1$ and the PR curve has a vertical segment at $\mathrm{Recall}=1$ (see the remark in [Appendix B](#sec-app-pr)).Since $\mathrm{Recall}=\mathrm{TPR}=r$ and $\mathrm{FPR}=G(r)$, the PR curve is given by$$\begin{align*}\mathrm{Precision}(r)& =\frac{\pi_1\,r}{\pi_1\,r+\pi_0\,G(r)}\\& =\Bigl(1+\frac{\pi_0}{\pi_1}\frac{G(r)}{r}\Bigr)^{-1}.\end{align*}$$Consequently, the PR curve is determined by $G$ together with the class prevalence $\pi_1$. This representation will also be useful for expressing average precision as an integral over recall.::: {style="font-size:90%; margin: 1em 0;"}| **Law of $Z$** | **$F(z)$** | **$G(r)$, $r=\mathrm{Recall}$** ||:---------------|:-----------|:--------------------------------|| **Gaussian** | $\Phi(z)$ | $\Phi\!\left(\frac{\Phi^{-1}(r)-\Delta}{\rho}\right)$ || **Cauchy** | $\Psi(z)$ | $\Psi\!\left(\frac{\Psi^{-1}(r)-\Delta}{\rho}\right)$ || **Logistic** | $\frac{1}{1+e^{-z}}$ | $\frac{r^{1/\rho}}{r^{1/\rho}+e^{\Delta/\rho}(1-r)^{1/\rho}}$ || **Uniform** $(-\frac12,\frac12)$ | $[z+\frac12]_0^1$ | $\left[\frac{r-\Delta-\frac{1-\rho}{2}}{\rho}\right]_0^1$ || **Gumbel** | $e^{-e^{-z}}$ | $1-e^{\left[-e^{-\frac{\Delta}{\rho}}(-\ln(1-r))^{\frac{1}{\rho}}\right]}$ || **Exp.** | $(1-e^{-z})_+$ | $\min\!\left(e^{-\Delta/\rho}r^{1/\rho},1\right)$ |: FPR as a function of recall, for common distributions. The precision is $\mathrm{Precision}(r)=\bigl(1+\tfrac{\pi_0}{\pi_1}\tfrac{G(r)}{r}\bigr)^{-1}$. {#tbl-distributions-pr .striped .hover}:::**Where the table comes from.** As for ROC, all the rows follow from a single identity (@thm-location-scale-pr): at the threshold with recall $r$,$$G(r)=1-F\Bigl(\frac{F^{-1}(1-r)+\Delta}{\rho}\Bigr),$$which simplifies to $G(r)=F\bigl((F^{-1}(r)-\Delta)/\rho\bigr)$ when $F$ is symmetric (@cor-symmetric-pr). The statements, the proofs, and the row-by-row computations are collected in [Appendix B](#sec-app-pr).**Visualization**The widget below lets you explore @thm-location-scale-pr interactively for the distribution described in @tbl-distributions-pr.Use the sliders on the left to set $\mu_0,\sigma_0$ (healthy class), $\mu_1,\sigma_1$ (sick class) and the **Prevalence** $\pi_1$, and drag the **Threshold** slider to move $t$.- The **middle panel** shows the two population densities plotted vertically: $X_1$ (sick) on the left in blue, $X_0$ (healthy) on the right in red. The horizontal black line marks the current threshold $t$; the darker-shaded region of each density is $\{X<t\}$ (the portion of that class predicted *negative* under the rule $\hat{Y}_t=\mathbb{1}\{X\ge t\}$) while the lighter region above the line is predicted *positive*.- The **right panel** shows the theoretical PR curve for the current $(\mu_0,\sigma_0,\mu_1,\sigma_1,\pi_1)$, with the dashed horizontal line $\mathrm{Precision}=\pi_1$ as the random-guessing reference (@thm-random-pr) and a marker at the operating point $(\mathrm{Recall}(t),\mathrm{Precision}(t))$ corresponding to the threshold you've selected, so you can see exactly which point on the curve a given threshold produces, and watch the curve itself deform as you change the means, standard deviations, their ratio $\rho=\sigma_0/\sigma_1$, or the prevalence $\pi_1$. Whatever the parameters, the curve ends at $(1,\pi_1)$ on the right.The title above the plot reports the current $\Delta=(\mu_1-\mu_0)/\sigma_1$, $\rho = \sigma_0/\sigma_1$ and $\pi_1$.::: {#fig-gaussian-pr-interactive}```{=html}<div id="gaussian-pr-interactive" class="gaussian-pr-interactive" style="display:flex; gap:10px; align-items:stretch; flex-wrap:nowrap; width:100%; max-width:min(895px, 100%); margin:0 auto; overflow-x:auto; font-family:'Inter', -apple-system, 'Segoe UI', Helvetica, Arial, sans-serif;"> <div class="gaussian-pr-controls" style="display:flex; flex-direction:column; justify-content:center; min-width:125px; flex:0 0 115px; border:1px solid rgba(0,0,0,0.15); border-radius:8px; padding:6px; font-size:9px; background:rgba(0,0,0,0.02);"> </div> <div class="gaussian-pr-plot" style="flex:1 1 auto; min-width:260px; display:flex; align-items:center; justify-content:center;"> </div></div><script type="module" src="gaussian_pr_interactive.js"></script>```**PR curve for classic distributions**:::**Note**: Visually, the recall is the ratio of the light blue area over the full blue area on the left, while the precision is the share of the predicted-positive mass that is blue: $\pi_1$ times the light blue fraction, divided by that quantity plus $\pi_0$ times the light red fraction.**Interpretation**$\Delta=(\mu_1-\mu_0)/\sigma_1$ and $\rho=\sigma_0/\sigma_1$ set the difficulty and shape of the underlying ROC curve, hence the function $G$; the prevalence $\pi_1$ then rescales the odds of precision.* **Odds form:** $\dfrac{\mathrm{Precision}}{1-\mathrm{Precision}}=\dfrac{\pi_1}{\pi_0}\cdot\dfrac{\mathrm{TPR}}{\mathrm{FPR}}$. Lowering $\pi_1$ with the ROC curve fixed lowers the precision at every recall level, and the PR curve is pulled toward the horizontal axis. At fixed $(\Delta,\rho)$ the ROC curve is the same for every $\pi_1$, but the PR curve is not.* **$\Delta=0,\ \rho=1$ (symmetric $F$):** $G(r)=r$, so $\mathrm{Precision}\equiv\pi_1$: the PR curve is the random-guessing baseline of @thm-random-pr. As in the ROC case, $\Delta=0$ alone is *not* enough when $\rho\ne1$.* **Right end of the curve, $\mathrm{Recall}\to1$:** $\mathrm{Precision}\to\pi_1$ for every $(\Delta,\rho,F)$, which is the universal endpoint $(1,\pi_1)$.* **Left end of the curve, $\mathrm{Recall}\to0$:** the limit is governed by the tail ratio $G(r)/r$ as $r\to0$. In the Gaussian case it is $1$ if $\rho<1$, or $\rho=1$ and $\Delta>0$; it is $\pi_1$ if $\rho=1$ and $\Delta=0$; and it is $0$ if $\rho>1$, or $\rho=1$ and $\Delta<0$. A more dispersed healthy class ($\rho>1$) therefore puts mostly healthy samples in the extreme upper tail.* **$\Delta\to\infty$** (fixed $\rho$): $G(r)\to0$ for each $r\in(0,1)$, so $\mathrm{Precision}(r)\to1$ and the curve approaches the ideal curve through $(1,1)$.Now we illustrate the empirical counterpart, when one samples elements from the distributions but without access to the true densities.::: {#fig-gaussian-pr-interactive-samples}```{=html}<div id="sampling-pr-interactive" class="sampling-pr-interactive" style="display:flex; gap:10px; align-items:stretch; flex-wrap:nowrap; width:100%; max-width: min(895px, 100%); margin:0 auto; overflow-x:auto; font-family:'Inter', -apple-system, 'Segoe UI', Helvetica, Arial, sans-serif;"> <div class="sampling-pr-controls" style="display: flex; flex-direction:column; justify-content: center; min-width:130px; flex:0 0 120px; border:1px solid rgba(0,0,0,0.15); border-radius:8px; padding:6px; font-size:9px; background:rgba(0,0,0,0.02);"> </div> <div class="sampling-pr-plot" style="flex:1 1 auto; min-width:260px; display:flex; align-items:center; justify-content:center;"> </div></div><script type="module" src="pr_sampling_explorer.js"></script>```**Empirical counterpart of PR curve for classic distributions.**:::**Note**: the jittering on the x-axis for the scatter plots is not meaningful but only here for visualization.### Average Precision (AP)**Average Precision (AP)** summarizes the precision–recall (PR) curve by integrating precision over recall, just as the area under the ROC curve (AUC) summarizes the ROC curve. Unlike ROC AUC, AP depends on the positive-class prevalence $\pi_1$; under random guessing, its population value is $\pi_1$.With $t_r$ and $G(r)=\mathrm{FPR}(t_r)$ as defined in the previous section, the precision at recall level $r$ is $\mathrm{Precision}(t_r)=\dfrac{\pi_1 r}{\pi_1 r+\pi_0 G(r)}$.:::: {#thm-ap}# Average precision as the area under the PR curveLet $X_0\sim\mathcal{L}(X\mid Y=0)$ and $X_1\sim\mathcal{L}(X\mid Y=1)$ be independent. Assume that the CDF $F_1$ of $X_1$ is continuous and strictly increasing wherever $0<F_1(t)<1$.Define average precision by$$\mathrm{AP}\triangleq \mathbb{E}\bigl[\mathrm{Precision}(X_1)\bigr].$$Then AP equals the area under the PR curve:$$\boxed{\mathrm{AP}=\int_0^1\frac{\pi_1 r}{\pi_1 r+\pi_0G(r)}\,dr.}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-ap')"}[▶]{#proof-arrow-thm-ap .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-ap .proof-content}Let $X_1'$ be an independent copy of $X_1$, and define$$R\triangleq 1-F_1(X_1).$$By the probability integral transform, $R\sim\mathrm{Uniform}(0,1)$. Moreover, since $X_1$ has a continuous distribution,$$\begin{aligned}R&=\mathbb{P}(X_1'\ge X_1\mid X_1)\\&=\mathrm{Recall}(X_1).\end{aligned}$$By the definition of $t_r$ and the strict monotonicity of $F_1$ on its nontrivial range, $X_1=t_R$ almost surely. Hence,$$\begin{aligned}\mathrm{Precision}(X_1)&=\frac{\pi_1R}{\pi_1R+\pi_0\mathbb{P}(X_0\ge X_1)}\\[4pt]&=\frac{\pi_1R}{\pi_1R+\pi_0G(R)}.\end{aligned}$$Since $R>0$ almost surely and $\pi_1>0$, the denominator is positive almost surely. Taking expectations and using $R\sim\mathrm{Uniform}(0,1)$ yields$$\begin{aligned}\mathrm{AP}&=\mathbb{E}\bigl[\mathrm{Precision}(X_1)\bigr]\\[4pt]&=\int_0^1\frac{\pi_1r}{\pi_1r+\pi_0G(r)}\,dr\\[4pt]&=\int_0^1\mathrm{Precision}(t_r)\,dr.\end{aligned}$$The last integral is the area under the PR curve, parametrized by recall. This establishes the result.:::::::Two remarks connect the AP with the ROC curve and the AUC.* **Dependence on the prevalence.** $G$ is the ROC curve read in the other direction, so it is fixed by the two conditional laws alone. The prevalence enters only through the weight $\pi_0/\pi_1$ in$$\mathrm{Precision}(r)=\Bigl(1+\frac{\pi_0}{\pi_1}\,\frac{G(r)}{r}\Bigr)^{-1}.$$The AP is therefore a functional of the ROC curve *and* of $\pi_1$.* **Comparison with the AUC.** By @cor-continuous-auc, $\mathrm{AUC}=\mathbb{P}(X_1>X_0)$, hence $\mathrm{AUC}=\int_0^1\bigl(1-G(r)\bigr)\,dr$, which does not depend on $\pi_1$. Both metrics average over recall levels $r$ uniformly, but they penalize the false positive rate differently: the AUC penalizes $G(r)$ itself, equally at every recall level, whereas the AP penalizes the ratio $G(r)/r$. A given false positive rate therefore costs more precision at *low* recall, so the AP rewards good behavior there (see the end of the interpretation above).Like the AUC, the AP has a natural empirical counterpart that only depends on the ranks of the samples.:::: {#thm-empirical-ap}# Empirical average precisionLet $X_{1,1},\dots,X_{1,n_1}$ and $X_{0,1},\dots,X_{0,n_0}$ be independent i.i.d. samples of the continuous variables $X_1$ and $X_0$, so that there are no ties almost surely. Let $X_{1,(1)}>\dots>X_{1,(n_1)}$ be the ordered $X_{1,i}$'s, and $\rho_k$ the rank of $X_{1,(k)}$ from the top of the pooled sample. Then the step-wise area under the empirical PR curve is$$\begin{align*}\widehat{\mathrm{AP}}& =\frac1{n_1}\sum_{k=1}^{n_1}\frac{k}{\rho_k}\\& =\frac1{n_1}\sum_{i=1}^{n_1}\widehat{\mathrm{Precision}}(X_{1,i}),\end{align*}$$the plug-in estimator of $\mathbb{E}[\mathrm{Precision}(X_1)]$.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-empirical-ap')"}[▶]{#proof-arrow-thm-empirical-ap .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-empirical-ap .proof-content}**Step 1: precision at the $k$-th positive sample.** Fix $k$ and let $t=X_{1,(k)}$. Since there are no ties, the positive samples $\ge t$ are exactly $X_{1,(1)},\dots,X_{1,(k)}$, so $\mathrm{TP}(t)=k$. The pooled samples $\ge t$ are exactly the $\rho_k$ samples ranked first to $\rho_k$-th from the top, so $\hat{\mathrm{P}}_t=\rho_k$. Hence$$\widehat{\mathrm{Precision}}(X_{1,(k)})=\frac{\mathrm{TP}(t)}{\hat{\mathrm{P}}_t}=\frac{k}{\rho_k}.$$**Step 2: the step-wise area.** The recall jumps by $1/n_1$ exactly when $t$ reaches a positive sample $X_{1,(k)}$, and the precision at that threshold is $k/\rho_k$. The step-wise area is the sum over the jumps of (jump size) $\times$ (precision at the jump), that is$$\widehat{\mathrm{AP}}=\sum_{k=1}^{n_1}\frac1{n_1}\cdot\frac{k}{\rho_k},$$which is the first form.**Step 3: the second form.** By Step 1, $\widehat{\mathrm{AP}}=\frac1{n_1}\sum_{k=1}^{n_1}\widehat{\mathrm{Precision}}(X_{1,(k)})$. The $X_{1,(k)}$ are the $X_{1,i}$ listed in decreasing order, so permuting the summands gives$$\widehat{\mathrm{AP}}=\frac1{n_1}\sum_{i=1}^{n_1}\widehat{\mathrm{Precision}}(X_{1,i}),$$which is the second form.**Step 4: plug-in.** Let $\hat{\pi}_y=n_y/(n_0+n_1)$. Then $\hat{\pi}_1\widehat{\mathrm{TPR}}(t)=\mathrm{TP}(t)/(n_0+n_1)$ and $\hat{\pi}_0\widehat{\mathrm{FPR}}(t)=\mathrm{FP}(t)/(n_0+n_1)$, so the formula of @thm-precision-formula, evaluated on these empirical quantities, gives$$\begin{align*}\frac{\hat{\pi}_1\,\widehat{\mathrm{TPR}}(t)}{\hat{\pi}_1\,\widehat{\mathrm{TPR}}(t)+\hat{\pi}_0\,\widehat{\mathrm{FPR}}(t)}& = \frac{\mathrm{TP}(t)}{\mathrm{TP}(t)+\mathrm{FP}(t)}\\& = \widehat{\mathrm{Precision}}(t).$$Since $\frac1{n_1}\sum_i\delta_{X_{1,i}}$ is the empirical law of $X_1$, the second form is $\mathbb{E}[\mathrm{Precision}(X_1)]$ with the law of $X_1$ replaced by its empirical law, and $\mathrm{Precision}$ by $\widehat{\mathrm{Precision}}$.:::::::In @fig-gaussian-pr-interactive-samples, the gray curve is the empirical PR curve, with consecutive points joined by straight segments for display only. The $\widehat{\mathrm{AP}}$ of @thm-empirical-ap is **not** the area under this drawn curve: it is the area of the staircase with height $k/\rho_k$ on the $k$-th recall step. This is what scikit-learn's `average_precision_score` computes. The trapezoidal rule gives a different value, because precision is not linear in the recall between two points. The gap shrinks as $n_1$ grows.Finally, we place the AP between two extreme values, which gives it the same kind of scale as the AUC has between $0$ and $1$.:::: {#thm-ap-range}# Range and reference values of the APUnder the assumptions of @thm-ap,$$\boxed{1+\frac{\pi_0}{\pi_1}\ln\pi_0\ \le\ \mathrm{AP}\ \le\ 1.}$$The upper bound is attained if and only if the classifier is perfect ($\mathrm{AUC}=1$), and the lower bound if and only if it is always wrong ($\mathrm{AUC}=0$). Moreover, for random guessing ($X_0\stackrel d=X_1$), $\mathrm{AP}=\pi_1$, and the lower bound is strictly smaller than $\pi_1$: an always-wrong classifier does worse than random guessing.:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-ap-range')"}[▶]{#proof-arrow-thm-ap-range .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-ap-range .proof-content}By @thm-ap, $\mathrm{AP}=\displaystyle\int_0^1\frac{\pi_1 r}{\pi_1 r+\pi_0G(r)}\,dr$, where $G(r)\in[0,1]$ is an FPR. The integrand is decreasing in $G(r)$, so both bounds follow from $0\le G\le 1$.*Upper bound (perfect classifier).* With $G(r)\ge0$, the integrand is at most $1$, so $\mathrm{AP}\le1$. Equality holds if and only if $G=0$ almost everywhere, that is, if and only if $\int_0^1 G(r)\,dr=0$. Since $\mathrm{AUC}=\int_0^1(1-G(r))\,dr$, this means $\mathrm{AUC}=1$: the **perfect classifier** of @thm-perfect-pr-separation.*Lower bound (always-wrong classifier).* With $G(r)\le1$, the integrand is at least $\frac{\pi_1 r}{\pi_1 r+\pi_0}$, so$$\mathrm{AP}\ \ge\ \int_0^1\frac{\pi_1 r}{\pi_1 r+\pi_0}\,dr=1+\frac{\pi_0}{\pi_1}\ln\pi_0 .$$Equality holds if and only if $G=1$ almost everywhere, that is, if and only if $\mathrm{AUC}=0$: the **always-wrong classifier** of @thm-always-wrong-pr, whose PR curve is exactly the arc $\frac{\pi_1 r}{\pi_1 r+\pi_0}$.*Random guessing.* If $X_0\stackrel d=X_1$, then $\mathrm{Precision}(t)=\pi_1$ for every $t$ where it is defined (@thm-random-pr). Hence $\mathrm{AP}=\pi_1$.*Comparison.* Writing $a=\pi_1$, the inequality $1+\frac{\pi_0}{\pi_1}\ln\pi_0<\pi_1$ is equivalent to $\ln(1-a)<-a$, which holds for every $a\in(0,1)$.:::::::**Reading the AP.** The AP is read against its baseline $\pi_1$: an AP of $0.3$ is excellent when $\pi_1=0.01$ but poor when $\pi_1=0.4$. APs are therefore not comparable across data sets with different prevalence, unlike AUCs.## Back to the COVID-19 use case### Data preprocessingThe file `supplementary_dataset2.csv` contains the sample-level data of the benchmark study: antibody assay measurements, sample-collection categories, and timing information such as the days since symptom onset.Our primary benchmark follows the study's cohort definition. Positives are laboratory-confirmed SARS-CoV-2 samples collected at least 20 days after symptom onset (only the latest qualifying sample is kept per participant), from the Oxford and NHSBT/Porton study populations. Negatives are the pre-pandemic samples of the Oxford Biobank. The preprocessing has three steps:1. select the study populations of the primary analysis;2. apply the eligibility criteria and keep the latest qualifying sample per participant;3. convert the assay measurements to numbers and keep the samples with complete results on all five assays.The resulting complete-case cohort has $n=1512$ observations, with $n_1=536$ positive and $n_0=976$ negative samples. These counts describe the composition of the benchmark, not the prevalence of infection in a clinical population.```{python}import numpy as npimport pandas as pdimport matplotlib# Colors of the figures: Set1 palette (positives and negatives)colors = matplotlib.colormaps["Set1"].colorsraw_df = pd.read_csv("supplementary_dataset2.csv", comment="#", na_values=["n/a", "NA", "NaN"],)# Drop the samples that are not SARS-CoV-2 cases (other coronaviruses,# other respiratory infections, convalescent or seasonal samples, ...)non_sars_cov2 = raw_df[~raw_df["samplecollection"].eq("obb_negatives")& raw_df["diseaseseverity"].str.contains("MERS|coronavirus|respiratory|convalescent|seasonal|other", case=False, na=False, )]raw_df = raw_df.drop(non_sars_cov2.index).copy()raw_df["covid_status"] = np.where( raw_df["samplecollection"].eq("obb_negatives"), "Negative", "Positive")assay_cols = ["abbott_result","diasorin_result","oxford_result","roche_result","siemens_result",]def assay_to_numeric(x):"""Convert a raw assay result to a number. Censored values are mapped to a finite value: "<3.80" -> 1.90 (half the detection limit), ">10.00" -> 10.00. """if pd.isna(x):return np.nan x =str(x).strip()try:if x.startswith("<"):returnfloat(x[1:].strip()) /2if x.startswith(">"):returnfloat(x[1:].strip())returnfloat(x)exceptValueError:return np.nan# 1. Study populations of the primary benchmarkdf = raw_df[ raw_df["samplecollection"].isin( ["obb_negatives", "oxford_positives", "porton_nhsbt_positives"] )].copy()df["dayssincesymptoms"] = pd.to_numeric(df["dayssincesymptoms"], errors="coerce")# 2. Eligibility# - all pre-pandemic Oxford Biobank samples are negative references;# - Oxford positives require >= 20 days since symptom onset;# - NHSBT positives are known to be >= 28 days post symptom onset.eligible_df = df[ df["covid_status"].eq("Negative")| df["samplecollection"].eq("porton_nhsbt_positives")| ( df["samplecollection"].eq("oxford_positives")& df["dayssincesymptoms"].ge(20) )].copy()# ... and keep the latest qualifying sample per participant# (missing participant IDs must remain distinct; missing delays are NHSBT: 28)eligible_df["participant_id"] = eligible_df["participantnum"].astype("string")missing = eligible_df["participant_id"].isna()eligible_df.loc[missing, "participant_id"] = ("missing_"+ eligible_df.index[missing].astype(str))eligible_df["sort_days"] = eligible_df["dayssincesymptoms"].fillna(28)benchmark_df = ( eligible_df.sort_values(["participant_id", "sort_days"], ascending=[True, False]) .drop_duplicates("participant_id", keep="first") .drop(columns=["participant_id", "sort_days"]) .copy())# 3. Numeric assay values; complete cases on the five assaysfor col in assay_cols: benchmark_df[col] = benchmark_df[col].map(assay_to_numeric)# Zero values cannot be shown on a log axis: use a small positive valuebenchmark_df["abbott_result"] = benchmark_df["abbott_result"].replace(0, 0.005)pcr_df = benchmark_df.dropna(subset=assay_cols).copy()print("Complete-case cohort:")print(pcr_df["covid_status"].value_counts())print("\nBy sample collection:")print(pcr_df.groupby(["covid_status", "samplecollection"]).size())# Assay labels and manufacturers' recommended thresholds (None: not available)dict_results_inv = {"abbott_result": "Abbott","diasorin_result": "DiaSorin","oxford_result": "Oxford","roche_result": "Roche","siemens_result": "Siemens",}dict_threshold = {"abbott_result": 1.4,"diasorin_result": 15.0,"oxford_result": None,"roche_result": 1.0,"siemens_result": 1.0,}```### Exporting data for the interactive figureThe interactive figure below runs client-side (Plotly.js), without a live Python kernel. We therefore export to JSON the sample values, the jitter offsets, the colors, and the ROC and PR curves of each assay, computed with the `roc` and `precision_recall` functions above.```{python}#| output: falseimport jsonimport matplotlib.colors as mcolorsfrom sklearn.metrics import auc, average_precision_scoredef export_roc_pr_data( df, dict_results_inv, dict_threshold, colors, seed=123, jitter=0.3, n_thresh=200):"""ROC / PR curves of each assay (on its non-missing observations).""" rng = np.random.RandomState(seed) df_pos_all = df[df["covid_status"].eq("Positive")] df_neg_all = df[df["covid_status"].eq("Negative")]# Horizontal jitter of the scatter panel (purely visual) x_pos = (jitter * (rng.rand(len(df_pos_all)) -0.5)).tolist() x_neg = (0.6+ jitter * (rng.rand(len(df_neg_all)) -0.5)).tolist() assays = {}for key, label in dict_results_inv.items(): dfa = df.loc[df[key].notna()] y_true = dfa["covid_status"].eq("Positive").values score = dfa[key].astype(float).values# Threshold range, valid on a log scale positive_scores = score[score >0] lower = positive_scores.min() /2iflen(positive_scores) else1e-6 upper = score.max() *2if score.max() >0else1.0if upper <= lower: upper = lower *2 ylims = [float(max(lower, 1e-12)), float(upper)] thresholds = np.logspace(np.log10(ylims[0]), np.log10(ylims[1]), num=n_thresh) tpr, fpr = roc(thresholds, y_true, score) precision, recall = precision_recall(thresholds, y_true, score)# AUC: trapezoidal rule on the ROC curve; AP: scikit-learn's# step-wise definition (see the AP section above) order = np.argsort(fpr) auc_value =float(auc(np.array(fpr)[order], np.array(tpr)[order])) ap_value =float(average_precision_score(y_true, score)) thr = dict_threshold[key] assays[label] = {"key": key,"pos": dfa.loc[y_true, key].astype(float).tolist(),"neg": dfa.loc[~y_true, key].astype(float).tolist(),"ylims": ylims,"threshold_init": float(np.median(score)),"threshold_reco": Noneif thr isNoneelsefloat(thr),"roc_fpr": fpr,"roc_tpr": tpr,"auc": auc_value,"pr_precision": precision,"pr_recall": recall,"ap": ap_value,"n_pos": int(y_true.sum()),"n_neg": int((~y_true).sum()), }# Best assay (largest AUC) first assays =dict(sorted(assays.items(), key=lambda kv: kv[1]["auc"], reverse=True))return {"x_pos": x_pos,"x_neg": x_neg,"colors": {"pos": mcolors.to_hex(colors[1]), "neg": mcolors.to_hex(colors[0])},"assays": assays,"default_assay": "DiaSorin"if"DiaSorin"in assays elsenext(iter(assays), None),"n_pos": int(len(df_pos_all)),"n_neg": int(len(df_neg_all)), }def subsample_positives(df, frac_pos=0.05, seed=123):"""Keep every negative sample, but only a fraction of the positives.""" pos = df[df["covid_status"].eq("Positive")] neg = df[df["covid_status"].eq("Negative")] pos_sub = pos.sample(n=max(1, round(len(pos) * frac_pos)), random_state=seed)return pd.concat([pos_sub, neg], ignore_index=True)# Full data, and a teaching example with a much rarer positive classfor filename, data in [ ("roc_data.json", pcr_df), ("roc_data_imbalanced.json", subsample_positives(pcr_df, frac_pos=0.05)),]: export = export_roc_pr_data(data, dict_results_inv, dict_threshold, colors)withopen(filename, "w") as f: json.dump(export, f)```### Interactive exploration: assay results, ROC curve and precision/recall curveThe widget below compares several assays at once. Tick one or more of them in the left-hand box: the first one you tick is the *primary* assay, and the threshold slider (log scale) acts on it. The three panels show:- **left:** the individual results of the primary assay, split by true status;- **top right:** the ROC curves of all the selected assays;- **bottom right:** their precision/recall curves, with the AP (**Average Precision**) in the legend.When the manufacturer provides a recommended threshold, it appears as a dotted grey line (left panel) and a grey cross on the curves ("Reco. point").::: {#fig-assays-interactive}```{=html}<div id="roc-interactive" class="roc-interactive" data-json="roc_data.json" data-layout="stacked" style="display:flex; gap:6px; align-items:stretch; flex-wrap:wrap; width:100%; max-width:840px; margin:0 auto; font-family:'Inter', -apple-system, 'Segoe UI', Helvetica, Arial, sans-serif;"> <div style="display:flex; flex-direction:column; justify-content:center; min-width:88px; flex:0 0 88px; border:1px solid rgba(0,0,0,0.15); border-radius:6px; padding:4px; font-size:8.5px; background:rgba(0,0,0,0.02);"> <div style="margin-bottom:2px; font-weight:600; font-size:8.5px;">Assay (click)</div> <div class="roc-assay-radios" style="font-size:8.5px; line-height:1.15; display:flex; flex-direction:column; gap:2px;"></div> <hr style="margin:4px 0; border-color:rgba(0,0,0,0.1);"> <div style="margin-bottom:2px; font-weight:600; font-size:8.5px;">Threshold</div> <input type="range" class="roc-threshold" min="0" max="1" step="0.001" value="0.5" style="width:100%; margin:0;"> <div class="roc-threshold-value" style="margin-top:2px; text-align:center; font-size:8px; color:#555;"></div> </div> <div class="roc-plot" style="display:flex; align-items:center; justify-content:center;"></div></div><script type="module" src="roc_interactive.js"></script>```**Assays on the full data** (536 positive and 976 negative samples).:::Now we rerun the same assays, with the same thresholds, but keep only 5% of the positive samples (the negatives are untouched). For any given assay, the **ROC curve barely moves**, while the **precision/recall curve** and the AP drop markedly. This is what the ray-slope argument above predicts: shrinking $\mathrm{P}$ raises the ratio $\mathrm{N}/\mathrm{P}$, which pushes the precision down at every recall level, while the ray slopes, hence the ROC curve, stay the same.::: {#fig-assays-interactive-imbalanced}```{=html}<div id="roc-interactive-imbalanced" class="roc-interactive" data-json="roc_data_imbalanced.json" data-layout="stacked" style="display:flex; gap:6px; align-items:stretch; flex-wrap:wrap; width:100%; max-width:840px; margin:0 auto; font-family:'Inter', -apple-system, 'Segoe UI', Helvetica, Arial, sans-serif;"> <div style="display:flex; flex-direction:column; justify-content:center; min-width:88px; flex:0 0 88px; border:1px solid rgba(0,0,0,0.15); border-radius:6px; padding:4px; font-size:8.5px; background:rgba(0,0,0,0.02);"> <div style="margin-bottom:2px; font-weight:600; font-size:8.5px;">Assay (click)</div> <div class="roc-assay-radios" style="font-size:8.5px; line-height:1.15; display:flex; flex-direction:column; gap:2px;"></div> <hr style="margin:4px 0; border-color:rgba(0,0,0,0.1);"> <div style="margin-bottom:2px; font-weight:600; font-size:8.5px;">Threshold</div> <input type="range" class="roc-threshold" min="0" max="1" step="0.001" value="0.5" style="width:100%; margin:0;"> <div class="roc-threshold-value" style="margin-top:2px; text-align:center; font-size:8px; color:#555;"></div> </div> <div class="roc-plot" style="display:flex; align-items:center; justify-content:center;"></div></div>```**Same assays, with only 5% of the positive samples kept** (same panels and thresholds as in @fig-assays-interactive).:::## {{< fa link title="link" >}} References::: {#refs}:::## Credit, sources and data- Paper of interest: [@fawcett2006introduction]- Paper with different measure for AUCROC/AUPRC [@chen2024commonly]- Original study article (Lancet): <https://doi.org/10.1016/S1473-3099(20)30634-4>- Dataset: [Head-to-head benchmark evaluation of five SARS-CoV-2 serology immunoassays on 1,500 samples (Figshare)](https://figshare.com/articles/dataset/Dataset_Head-to-head_benchmark_evaluation_of_the_sensitivity_and_specificity_of_five_immunoassays_for_SARS-CoV-2_serology_on_1500_samples/12622172).- Direct download: [supplementary_dataset2.xlsx in Figshare](https://ndownloader.figshare.com/files/23736068)## Appendix A: ROC curves for location-scale families {#sec-app-roc}This appendix gathers the computations behind @tbl-distributions, with the notation of the main text: $X_y=\sigma_yZ_y+\mu_y$, where $Z_0,Z_1$ are i.i.d. with CDF $F$, $\Delta=(\mu_1-\mu_0)/\sigma_1$, $\rho=\sigma_0/\sigma_1$, $\Phi$, $[x]_0^1$ and $(x)_+$.### ROC curve for location-scale family:::: {#thm-location-scale-roc}# Location-scale ROC curveLet $Z_0,Z_1$ be i.i.d. real-valued random variables with common CDF $F$, continuous and strictly increasing on $\{z\in\mathbb{R}:0<F(z)<1\}$, so that $F^{-1}:(0,1)\to\mathbb{R}$ is well defined. Fix $\mu_0,\mu_1\in\mathbb{R}$, $\sigma_0,\sigma_1>0$, and let $$X_0\triangleq \sigma_0Z_0+\mu_0, \quad X_1\triangleq \sigma_1Z_1+\mu_1$$be independent. Writing $\Delta\triangleq (\mu_1-\mu_0)/\sigma_1$, $\rho\triangleq \sigma_0/\sigma_1$, for every $\mathrm{FPR}\in (0,1)$,$$\mathrm{TPR}=1-F\Bigl(\rho\,F^{-1}\bigl(1-\mathrm{FPR}\bigr)-\Delta\Bigr).$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-location-scale-roc')"}[▶]{#proof-arrow-thm-location-scale-roc .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-location-scale-roc .proof-content}By definition,$$\begin{align*}\mathrm{FPR}(t)& =\mathbb{P}(X_0\ge t)\\& =\mathbb{P}\left(Z_0\ge\frac{t-\mu_0}{\sigma_0}\right)\\& =1-F\left(\frac{t-\mu_0}{\sigma_0}\right).\end{align*}$$Whenever $\mathrm{FPR}(t)\in(0,1)$, i.e., $1-\mathrm{FPR}(t)\in(0,1)$, $F^{-1}$ is well defined there and$$\frac{t-\mu_0}{\sigma_0}=F^{-1}\bigl(1-\mathrm{FPR}(t)\bigr).$$Likewise,$$\mathrm{TPR}(t)=1-F\!\left(\frac{t-\mu_1}{\sigma_1}\right),$$and$$\begin{align*}\frac{t-\mu_1}{\sigma_1}& =\frac{t-\mu_0}{\sigma_1}-\frac{\mu_1-\mu_0}{\sigma_1}\\& =\frac{\sigma_0}{\sigma_1}\cdot\frac{t-\mu_0}{\sigma_0}-\Delta \\& =\rho\,F^{-1}\bigl(1-\mathrm{FPR}(t)\bigr)-\Delta.\end{align*}$$Substituting this expression gives the result.The outer $F$ requires no monotonicity beyond its ordinary CDF definition, since it is applied to a real number rather than inverted.:::::::### Corollary: symmetric location-scale ROC:::: {#cor-symmetric-roc}# Symmetric location-scale ROC curveUnder the hypotheses of @thm-location-scale-roc, suppose in addition that $F$ is symmetric:$$F(-x)=1-F(x)\quad \forall x\in\mathbb{R}.$$Then for every $\mathrm{FPR}\in(0,1)$,$$\mathrm{TPR}=F\bigl(\Delta+\rho\,F^{-1}(\mathrm{FPR})\bigr).$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('cor-symmetric-roc')"}[▶]{#proof-arrow-cor-symmetric-roc .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-cor-symmetric-roc .proof-content}Fix $\mathrm{FPR}\in(0,1)$ and let$$x\triangleq F^{-1}(\mathrm{FPR}),$$so that $F(x)=\mathrm{FPR}$. By symmetry,$$F(-x)=1-F(x)=1-\mathrm{FPR}.$$Since $1-\mathrm{FPR}\in(0,1)$ as well, uniqueness of $F^{-1}$ on $(0,1)$ gives$$F^{-1}(1-\mathrm{FPR})=-x=-F^{-1}(\mathrm{FPR}).$$Substituting into @thm-location-scale-roc,$$\begin{aligned}\mathrm{TPR}&=1-F\Bigl(-\rho\,F^{-1}(\mathrm{FPR})-\Delta\Bigr)\\&=1-\Bigl(1-F\bigl(\rho\,F^{-1}(\mathrm{FPR})+\Delta\bigr)\Bigr)\\&=F\bigl(\Delta+\rho\,F^{-1}(\mathrm{FPR})\bigr),\end{aligned}$$where the middle step uses symmetry once more, at the point $-\rho F^{-1}(\mathrm{FPR})-\Delta$.:::::::### Applications: the symmetric rowsEach of the following is an immediate instance of @cor-symmetric-roc: verify that $F$ is symmetric, write down $F^{-1}$, and substitute.**Gaussian.** With $\Phi$ the standard normal CDF,$$\Phi(x)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{x}e^{-u^2/2}\,du,$$$F=\Phi$ satisfies $\Phi(-x)=1-\Phi(x)$ by construction. Substituting in @cor-symmetric-roc gives$$\mathrm{TPR}=\Phi\bigl(\Delta+\rho\,\Phi^{-1}(\mathrm{FPR})\bigr).$$**Cauchy.** The standard Cauchy CDF is $F(x)=\tfrac12+\tfrac1\pi\tan^{-1} x$. Since $\tan^{-1}$ is odd, $F(-x)=1-F(x)$.Solving $$\tfrac12+\tfrac1\pi\tan^{-1}(F^{-1}(p))=p$$ gives$$F^{-1}(p)=\tan\bigl(\pi(p-\tfrac12)\bigr).$$Substituting gives the Cauchy entry in the @tbl-distributions.**Logistic.** The logistic CDF is $F(x)=\dfrac{1}{1+e^{-x}}$. It is symmetric since $F(-x)=1-F(x)$, and $F^{-1}(p)=\ln\dfrac{p}{1-p}$. Therefore,$$\begin{align*}\mathrm{TPR}& = F\Bigl(\Delta+\rho\ln\tfrac{\mathrm{FPR}}{1-\mathrm{FPR}}\Bigr)\\& = \frac{1}{1+e^{-\Delta}\left(\tfrac{1-\mathrm{FPR}}{\mathrm{FPR}}\right)^\rho}.\end{align*}$$Multiplying numerator and denominator by $e^\Delta\mathrm{FPR}^\rho$ gives the entry in @tbl-distributions.**Uniform $(-\tfrac12,\tfrac12)$.** The CDF is obtained by clipping the shifted variate,$$F(z)=\bigl[z+\tfrac12\bigr]_0^1,$$which is symmetric on $\mathbb{R}$, with $F^{-1}(p)=p-\tfrac12$ on $(0,1)$.By the theorem, $\mathrm{TPR}=F(z_1)$ where $z_1=\Delta+\rho z_0$ and $z_0=F^{-1}(\mathrm{FPR})=\mathrm{FPR}-\tfrac12$. Hence$$\begin{align*}\mathrm{TPR}& = F\bigl(\Delta+\rho\bigl(\mathrm{FPR}-\tfrac12\bigr)\bigr)\\& = \Bigl[\Delta+\tfrac{1-\rho}{2}+\rho\,\mathrm{FPR}\Bigr]_0^1 .\end{align*}$$### The remaining rows: Gumbel and Exponential (not symmetric)**Gumbel.** The Gumbel-max CDF is $F(x)=\exp(-e^{-x})$.Its inverse is $F^{-1}(p)=-\ln(-\ln p)$, so with the previous theorem, $1-\mathrm{TPR}=F\left(\rho \ln\left(\frac{1}{-\ln(1-\mathrm{FPR})}\right) - \Delta\right)$.Setting $w\triangleq -\ln(1-\mathrm{FPR})>0$, the argument of $F$ is $-\rho\ln w-\Delta$, and$$\begin{align*}F(-\rho\ln w-\Delta)& = \exp\bigl(-e^{\rho\ln w+\Delta}\bigr)\\& = \exp\bigl(-e^\Delta w^\rho\bigr),\end{align*}$$hence$$\mathrm{TPR}(\mathrm{FPR})=1-e^{\Bigl(-e^\Delta\bigl(-\ln(1-\mathrm{FPR})\bigr)^\rho\Bigr)}.$$**Exponential.** The exponential CDF is $F(z)=(1-e^{-z})_+$. Its inverse on $(0,1)$ is $F^{-1}(p)=-\ln(1-p)$, so, with @thm-location-scale-roc,$$\mathrm{TPR}(t)=1-F\bigl(\rho\,(-\ln\mathrm{FPR})-\Delta\bigr).$$When $\rho(-\ln\mathrm{FPR})-\Delta\ge0$ (i.e. $\mathrm{FPR}\le e^{-\Delta/\rho}$), $F(x)=1-e^{-x}$ gives $1-F(\cdot)=e^{-(\rho(-\ln\mathrm{FPR})-\Delta)}=e^\Delta\mathrm{FPR}^\rho$; otherwise the argument is negative, $F\equiv0$ there, and $\mathrm{TPR}=1$. Hence$$\boxed{\mathrm{TPR}(\mathrm{FPR})=\min\bigl(e^\Delta\mathrm{FPR}^\rho,\,1\bigr).}$$## Appendix B: PR curves for location-scale families {#sec-app-pr}This appendix gathers the computations behind @tbl-distributions-pr, with the notation of Appendix A, the prevalence $\pi_1$, and $G(r)=\mathrm{FPR}(t_r)$ as defined in the main text.### PR curve for location-scale family:::: {#thm-location-scale-pr}# Location-scale PR curveUnder the hypotheses of @thm-location-scale-roc, let $\pi_1\in(0,1)$ and $\pi_0=1-\pi_1$. Then for every $r\in(0,1)$ there is a unique threshold $t_r$ with $\mathrm{Recall}(t_r)=r$, and$$\begin{align*}\mathrm{FPR}(t_r)& = 1-F\Bigl(\tfrac{F^{-1}(1-r)+\Delta}{\rho}\Bigr)\\\mathrm{Precision}(t_r)& = \frac{\pi_1\,r}{\pi_1\,r+\pi_0\,G(r)}\end{align*}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('thm-location-scale-pr')"}[▶]{#proof-arrow-thm-location-scale-pr .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-thm-location-scale-pr .proof-content}**Threshold.** Since $\mathrm{Recall}(t)=\mathrm{TPR}(t)=1-F\bigl(\tfrac{t-\mu_1}{\sigma_1}\bigr)$, the equation $\mathrm{Recall}(t)=r$ reads $F\bigl(\tfrac{t-\mu_1}{\sigma_1}\bigr)=1-r\in(0,1)$. As $F$ is strictly increasing on $\{0<F<1\}$, its unique solution is$$t_r=\mu_1+\sigma_1F^{-1}(1-r).$$**FPR.** At $t_r$,$$\begin{align*}\frac{t_r-\mu_0}{\sigma_0}& =\frac{\sigma_1F^{-1}(1-r)+\mu_1-\mu_0}{\sigma_0}\\& = \frac{F^{-1}(1-r)+\Delta}{\rho},\end{align*}$$so $\mathrm{FPR}(t_r)=1-F\bigl(\tfrac{t_r-\mu_0}{\sigma_0}\bigr)=G(r)$.**Precision.** By definition,$$\mathrm{Precision}(t)=\frac{\pi_1\,\mathrm{TPR}(t)}{\pi_1\,\mathrm{TPR}(t)+\pi_0\,\mathrm{FPR}(t)},$$whose denominator is at least $\pi_1 r>0$ at $t=t_r$. Substituting $\mathrm{TPR}(t_r)=r$ and $\mathrm{FPR}(t_r)=G(r)$ gives the second equality.:::::::### Corollary: symmetric location-scale PR:::: {#cor-symmetric-pr}# Symmetric location-scale PR curveUnder the hypotheses of @thm-location-scale-pr, suppose in addition that $F(-x)=1-F(x)$ for every $x\in\mathbb{R}$. Then for every $r\in(0,1)$,$$\boxed{G(r)=F\Bigl(\frac{F^{-1}(r)-\Delta}{\rho}\Bigr).}$$:::::::: {.proof-container}::: {.proof-header onclick="toggleProof('cor-symmetric-pr')"}[▶]{#proof-arrow-cor-symmetric-pr .proof-arrow} [Proof]{.proof-title}:::::: {#proof-content-cor-symmetric-pr .proof-content}Fix $r\in(0,1)$ and let $z\triangleq F^{-1}(r)$, so that $0<F(z)=r<1$.**Step 1: $F^{-1}(1-r)=-z$.** By symmetry, $F(-z)=1-F(z)=1-r\in(0,1)$. Hence $-z$ lies in $\{0<F<1\}$, where $F$ is strictly increasing, and $-z$ is the unique point of this set with $F=1-r$. That is, $F^{-1}(1-r)=-z$.**Step 2: simplify $G$.** Put $x\triangleq\dfrac{z-\Delta}{\rho}$. By Step 1, the argument of $F$ in the definition of $G(r)$ is$$\frac{F^{-1}(1-r)+\Delta}{\rho}=\frac{-z+\Delta}{\rho}=-x.$$Applying symmetry at the point $x$,$$\begin{align*}G(r)& = 1-F(-x)\\& = 1-\bigl(1-F(x)\bigr)& = F(x)\\& = F\Bigl(\frac{F^{-1}(r)-\Delta}{\rho}\Bigr).\end{align*}$$:::::::### Applications: the symmetric rowsEach is an immediate instance of @cor-symmetric-pr: verify that $F$ is symmetric, write down $F^{-1}$, and substitute into $G$.**Gaussian.** $F=\Phi$ satisfies $\Phi(-x)=1-\Phi(x)$, so$$G(r)=\Phi\Bigl(\frac{\Phi^{-1}(r)-\Delta}{\rho}\Bigr).$$**Cauchy.** $F(x)=\tfrac12+\tfrac1\pi\tan^{-1}x$ is symmetric since $\tan^{-1}$ is odd, with $F^{-1}(p)=\tan\bigl(\pi(p-\tfrac12)\bigr)$. Hence$G(r)=\tfrac12+\tfrac1\pi\tan^{-1}\!\Bigl(\frac{\tan(\pi(r-\frac12))-\Delta}{\rho}\Bigr).$**Logistic.** $F(x)=\dfrac{1}{1+e^{-x}}$ is symmetric, with $F^{-1}(p)=\ln\dfrac{p}{1-p}$. Therefore$$\begin{aligned}G(r)&=\frac{1}{1+\exp\bigl(-\tfrac{1}{\rho}\bigl(\ln\tfrac{r}{1-r}-\Delta\bigr)\bigr)}\\&=\frac{1}{1+e^{\Delta/\rho}\bigl(\tfrac{1-r}{r}\bigr)^{1/\rho}}\\&=\frac{r^{1/\rho}}{r^{1/\rho}+e^{\Delta/\rho}(1-r)^{1/\rho}}.\end{aligned}$$**Uniform $(-\tfrac12,\tfrac12)$.** $F(z)=[z+\tfrac12]_0^1$ is symmetric, with $F^{-1}(p)=p-\tfrac12$ on $(0,1)$. Hence$$\begin{align*}G(r)& = F\Bigl(\frac{r-\frac12-\Delta}{\rho}\Bigr)\\& = \Bigl[\frac{r-\Delta-\frac{1-\rho}{2}}{\rho}\Bigr]_0^1 .\end{align*}$$### The remaining rows: Gumbel and Exponential (not symmetric)Here we use @thm-location-scale-pr directly.**Gumbel.** $F(x)=\exp(-e^{-x})$ and $F^{-1}(p)=-\ln(-\ln p)$, so $F^{-1}(1-r)=-\ln(-\ln(1-r))$. The argument of $F$ in $G(r)$ is $a\triangleq\tfrac{\Delta-\ln(-\ln(1-r))}{\rho}$, and$$\begin{align*}F(a)& = \exp\bigl(-e^{-a}\bigr)\\& =e^{\Bigl(-\bigl(e^{-\Delta}(-\ln(1-r))\bigr)^{1/\rho}\Bigr)}.\end{align*}$$Hence,$$G(r)=1-e^{\Bigl(-e^{-\Delta/\rho}\bigl(-\ln(1-r)\bigr)^{1/\rho}\Bigr)}.$$**Exponential.** $F(z)=(1-e^{-z})_+$ and $F^{-1}(p)=-\ln(1-p)$, so $F^{-1}(1-r)=-\ln r$ and the argument of $F$ is $a\triangleq\tfrac{\Delta-\ln r}{\rho}$. If $a\ge0$, then $1-F(a)=e^{-a}=e^{-\Delta/\rho}r^{1/\rho}$. If $a<0$, then $F(a)=0$ and $G(r)=1$. Hence$$\boxed{G(r)=\min\bigl(e^{-\Delta/\rho}\,r^{1/\rho},\,1\bigr).}$$*Remark (bounded-below supports).* For the Uniform and Exponential laws, recall $1$ can be reached at a finite threshold, so the plotted curve $r\mapsto(r,\mathrm{Precision}(r))$ stops before $r=1$. The remaining part of the PR curve is the vertical segment at $\mathrm{Recall}=1$, from $\mathrm{Precision}(1^-)$ down to the universal endpoint $(1,\pi_1)$ (reached as $t\to-\infty$).
