Statistics, and how they are checked

Principles

• The unit is the biological sample. Technical replicates are averaged on the Cq scale before anything is tested, so each sample contributes one value and n counts biological replicates. Wells are never counted as replicates.
• The test is fixed by the design, before the data are seen. Choosing a test by first testing its assumptions distorts its error rates and gains nothing [19], and at the sample sizes of a qPCR experiment a normality test has almost no power anyway. The checks are shown next to the result. They never switch the test.
• Two-sided tests at α = 0.05.
• Every comparison comes with its size. A ratio and its 95% confidence interval, not a P value alone.
• From three groups, P values are adjusted for the number of comparisons made, and their intervals widened to match.

Why log2

A fold change is a ratio of exponentials of Cq values. The measurement noise of qPCR is roughly additive on the Cq scale, so on the fold-change scale it is multiplicative: fold changes are skewed, bounded by zero, and their spread grows with their mean. That breaks the equal-variance assumption of a t-test or an ANOVA by construction. On log2 the same noise is additive and roughly symmetric, which is what those tests expect.

So the tests run on log2(fold change) by default, the same choice as the lognormal option of GraphPad Prism. Two consequences follow. Every estimate comes back as a ratio of geometric means, with an interval that is symmetric on log2 and therefore asymmetric around the ratio. And the fold changes themselves are untouched: the calibrator stays the arithmetic mean of the control group, as the method specifies [1], and dividing every sample by the same constant changes no P value. Rank tests give the same P value on both scales, since log2 preserves order. Tests on the untransformed fold change remain available as an option.

Which test runs

Two groups: Welch. Welch’s t-test [16] does not assume the two groups share a variance, and when they do, with equal group sizes, it loses almost nothing against Student’s test [18]. One default that holds either way.

Three groups and up: ordinary ANOVA, then Dunnett. This is Prism’s default, and the reason is degrees of freedom. With three samples per group, pooling the variance of all groups gives the comparisons 6 degrees of freedom where a comparison using only its own two groups would have at most 4, and on log2 the equal-variance assumption behind the pooling is a reasonable one. When groups are known to differ in spread, Welch’s ANOVA [17] followed by Dunnett’s T3 [22] drops that assumption, at the price of those degrees of freedom.

To change the test, the scale or the comparisons, open the Assumptions tab of the batch and click Apply & re-analyse. The choice is stored with the batch and named in the PDF, the exports and the Methods paragraph. Tests that do not fit the design are not offered: there is no paired test without pairs.

Multiple comparisons

From three groups the omnibus test says whether the groups differ somewhere. The comparisons that follow say where, and their P values are adjusted for how many were made, so the chance of a false positive across the whole family stays at 5%. Two families are offered:

• Each group against the calibrator (the default): k − 1 comparisons. After an ordinary ANOVA this is Dunnett’s test [20], which uses the correlation between comparisons that share the calibrator and is therefore less conservative than a Bonferroni correction.
• Every pair of groups: k(k − 1)/2 comparisons, with Tukey’s test in its Tukey-Kramer form for unequal group sizes [21].

Welch’s ANOVA is followed by Dunnett’s T3 for either family; Kruskal-Wallis and Friedman by Dunn’s test [23], Bonferroni-adjusted as Prism does; the repeated-measures ANOVA by paired t-tests with the Šidák correction, each comparison using only its own two columns, which is what the Geisser-Greenhouse correction calls for. Every family comes with simultaneous confidence intervals, except Dunn’s test, which has none.

The comparisons are computed whatever the omnibus P value, as in Prism: a multiplicity-adjusted P value is valid on its own. On the charts, a comparison against the calibrator is always drawn, “ns” included, so the failures are as visible as the successes. When every pair is compared, only the significant ones are drawn, or a five-group chart would carry ten brackets. The stars follow GraphPad’s convention: ns P > 0.05, * P ≤ 0.05, ** P ≤ 0.01, *** P ≤ 0.001, **** P ≤ 0.0001.

Estimates and intervals

Under each chart, every comparison reads like this: KO vs WT 2.40× [1.62, 3.55] P = 0.0015. The ratio is the geometric mean of KO divided by the geometric mean of WT, and the bracket is its 95% confidence interval, adjusted when it belongs to a family. It is the most useful number on the page.

Rank tests report the Hodges-Lehmann estimate [25], the median of all the pairwise differences, with its exact interval. That interval can only take a few levels at small n and cannot always reach 95%: at three samples per group the closest is 90%, and Ampliova prints the level it actually reached. On the untransformed scale the estimates are differences of mean fold change instead of ratios.

Paired designs

A design is paired when the same animal, donor, patient or culture is measured in each condition. Give those samples a common pair ID in the confirmation step of the plate wizard (or when editing a plate): “M1” for mouse 1 in every group, for example. A pair ID given to a sample on one plate is carried to its wells on later plates.

• Only complete pairs, with one sample in every group, enter a paired test; the others are listed as left out.
• Two samples of one group under the same pair ID, or one sample under two pair IDs on different plates, stop the test: choosing between them would be inventing the pairing.
• Without pair IDs the design is unpaired. Ampliova never pairs samples by name or by order.
• Connect matched samples, in the figure settings, draws a line through each pair on dot and line charts.

Two-factor designs

A two-factor batch crosses two conditions, genotype and treatment for example, with the group field holding the first factor and a second field the other. The engine fits a fixed-effects two-way ANOVA with Type III sums of squares, as a regression on sum-to-zero codes [30], which is how Prism handles unequal cell sizes: the result does not depend on which factor is listed first. The interaction is reported first, because when it is significant the main effects cannot be read on their own.

The second factor is then compared within each level of the first (treated against vehicle in each genotype, say), Šidák-adjusted over all those comparisons, with the ratio and interval of each. The analysis needs at least one sample in every combination of the two factors and a replicate in at least one. It is not a repeated-measures or mixed model.

Checking assumptions

The Assumptions tab shows, for each gene, the residuals the test assumed normal: each sample’s deviation from its group mean, pooled across groups (the differences within pairs for a paired test, the deviations from each cell mean for a two-way ANOVA). They are drawn as a normal QQ plot with the same plotting positions and reference line as R’s qqnorm() and qqline(), so the plot matches one a reviewer would draw.

Next to it, Shapiro-Wilk [26] on those residuals, and the Brown-Forsythe test of equal variances [27] (Levene’s test centred on the median). Both are on the scale the test ran on, with a toggle to see the other. Read them as descriptions: with a dozen residuals Shapiro-Wilk rarely rejects, so a high P value is weak evidence of normality, and a low one on log2 data usually points at a single aberrant sample worth inspecting rather than at a different test.

Small samples

Two biological samples per group can be tested, but leave a test almost no degrees of freedom: the MIQE checklist warns below three. Rank tests have a harder floor. Their exact P value can only take a few values, and at small n none of them is below 0.05:

Below those sizes a rank test cannot come out significant whatever the data, which is one more reason the default is not chosen from a normality check. Ampliova warns before you apply such a test and under the result.

Many genes at once

Each gene is tested on its own. When a batch tests dozens of genes, an RT² Profiler array for instance, some will pass P < 0.05 by chance alone. The engine therefore also computes, for every batch, the Benjamini-Hochberg q-value of each gene across the batch [28], which controls the expected proportion of false discoveries among the genes called significant. The RT² results table, its figure filter and the shared results page all select genes on q; the volcano plot keeps the raw P value on its axis, as volcano plots do. The per-gene tests and their comparisons are unchanged.

Error bars

By default a bar is the arithmetic mean of the biological samples’ fold changes with its standard error, and every sample is drawn as a point. The figure settings offer the standard deviation, a 95% confidence interval, the geometric mean with its geometric SD or 95% CI, and the median with its interquartile range, range or 95% confidence interval. The median interval uses the order statistics, as Prism does, and needs six samples to reach 95%; below that the bars span the range and the legend says so.

The legend under every figure names what the bars show. Error bars do not test anything: overlapping bars can hide a significant difference and separate ones a non-significant one. The brackets carry the tests.

Validation against R

The statistics live in one module of the analysis engine, and a test suite compares its output with R on fixed datasets: two groups of equal and unequal sizes, ties, pairs, three to five groups, repeated measures, balanced and unbalanced two-factor designs. 1,114 values are checked, from every statistic and degree of freedom to every P value, estimate and confidence limit, against R 4.5.1 with car 3.1.5, emmeans 2.0.4, mvtnorm 1.4.2 and dunn.test 1.3.7. They agree to 1e-7 where R’s value is exact and to 2e-6 where it is itself a numerical integral.

• t-tests, rank tests with their Hodges-Lehmann intervals, one-way and Welch ANOVA, Kruskal-Wallis, Friedman: base R.
• Type III sums of squares, Brown-Forsythe and the Geisser-Greenhouse correction: car.
• Dunnett’s test and its critical values: mvtnorm. Tukey: base R’s TukeyHSD.
• Dunnett’s T3: an independent integration of the studentized maximum modulus, itself checked against mvtnorm.
• Two-way simple effects with Šidák: emmeans. Dunn’s test: dunn.test. Benjamini-Hochberg: p.adjust.

The engine runs on SciPy 1.14.1 and NumPy 1.26.4 [32], pinned to the versions the suite validates, and computes Dunnett’s and T3’s distributions by deterministic quadrature, so the same data give the same P value every time. The Methods paragraph names the versions a batch was analysed with. Batches analysed before these checks existed keep their stored result until they are re-analysed; the results page says when a batch predates them. See also the Methods paragraph.