Choose the Right Test

Start with two questions: what kind of data do you have, and what are you trying to find out? The answers determine which statistical test is appropriate.

Step 1: What is your research goal?

Group comparison

How many groups?

Two groups

Welch's t-test is the default for independent two-group comparisons. It is appropriate whether or not variances are equal, and variance checks are shown for interpretation and disclosure rather than to determine which t-test to run.

Three or more groups

For repeated measurements from the same subjects across three or more conditions, Licklider selects repeated measures ANOVA when all groups pass the normality test, or Friedman's test when any group fails. Both are selected automatically based on the normality check result.

When pairwise comparisons are requested after ANOVA, a multiple comparisons correction (Tukey HSD, Bonferroni, or Holm) is required. Licklider will ask which to use if one has not been specified.

For factorial designs with two grouping variables — see Two-Way ANOVA.

Mixed design (between x within)

When the study includes both a between-subjects factor and a within-subjects factor — for example, treatment groups measured across multiple time points — Licklider selects Mixed ANOVA.

Mixed ANOVA requires the study design to be explicitly identified as mixed during data setup. It is not selected from table structure alone.

How Licklider determines pairing

Licklider looks for an ID or pair column in your data. When one is present, it checks whether pairing is consistent across conditions and asks you to confirm the design before running the analysis. If no ID column is present, the design is treated as independent.

Licklider cannot infer every design fact from the table alone. Hidden nesting, repeated observations without a declared ID, and paired structures that are not encoded explicitly may all require your input. If those details are wrong, the suggested test can be wrong too.

If Licklider's assessment does not match your experimental design, tell it in the Chat.

How Licklider uses normality

Shapiro-Wilk results are one input into the test suggestion, not a hard switch. Licklider shows normality check results in the Inspector and uses them to inform its suggestion. If the suggestion does not fit your analysis plan, you can redirect it in the Chat.

Association and regression

For scatter and regression charts, Licklider automatically calculates both Pearson and Spearman correlations. The primary correlation is selected based on the normality of the regression residuals.

For more detail on each — see Regression and Modeling.

Nonlinear and specialized

For more detail — see Non-linear Regression and IC50/4PL, Repeated Measures and Mixed Models.

Survival analysis

For more detail — see Kaplan-Meier Analysis, Cox Proportional Hazards Regression.

Categorical association

For contingency tables, Licklider runs the chi-square test of independence. For 2x2 tables, Fisher's exact test is also run automatically. For larger tables (3x2, 2x3, 3x3, etc.) where expected cell counts are small, the Freeman-Halton extension of Fisher's exact test is available and runs alongside the chi-square result.

Note on small expected counts: When expected cell counts fall below 5 in a 2x2 table, or when the sample is small enough that the chi-square approximation is unreliable, Fisher's exact test is the more appropriate choice. The Freeman-Halton form extends the same exact-conditional logic to larger tables and uses a hybrid algorithm (exact enumeration for small tables, Monte Carlo for larger ones) so that the calculation stays tractable.

Both Fisher results (2x2 and the Freeman-Halton R-by-C form) are run as part of the chi-square workflow and appear in the Inspector alongside the chi-square result; Fisher is not available as a standalone method selection.

For more detail — see Fisher's Exact Test.

Sample size planning

For more detail — see Power Analysis and Sample Size.

Quality checks that affect test selection

Licklider runs several checks alongside test selection that may affect or override the initial suggestion:

• Pairing check — confirms that pairing is valid and consistent before a paired test is run
• Pseudoreplication check — detects cases where observations that appear independent actually share a biological source, which would inflate false positive rates in a standard group comparison
• Multiple comparisons — when pairwise tests are requested across three or more groups, a correction method must be confirmed before results are shown
• Batch confounding — if batch and group are confounded, the test result may reflect batch effects rather than treatment effects

These checks improve the suggestion, but they do not replace study-design judgment. If the dataset structure is misdeclared or incomplete, Licklider may still suggest a test that looks technically plausible while reflecting the wrong experimental design.

For more detail on each — see Normality and Homoscedasticity, Pseudoreplication Detection, Paired vs Unpaired Guard.

Project-wide consistency

Licklider also monitors test-selection consistency across all figures in a project. If two figures expose the same comparable grouping signals and assumption outcomes but use different primary test choices, the Project Audit flags that difference for review. For details, see Project Statistical Policy and Consistency Audit.

Adjusting Licklider's suggestion

Licklider's test selection is a starting point. If the suggested test does not match your experimental design or pre-registered analysis plan, tell Licklider in the Chat:

• "This is a paired design — use a paired t-test"
• "Use Student's t-test"
• "These groups are not paired"
• "Use Spearman correlation"

Manual overrides are recorded in the analysis record. This matters because Licklider cannot determine from the table alone whether the intended analysis is confirmatory or exploratory, whether a design factor was pre-specified, or whether scientific context requires a different method.

What this page does not cover

• Detailed method documentation for each test — see Methods
• How figures are chosen for different data structures — see Choose the Right Figure
• Common workflows end to end — see Common Workflows by Dataset Shape

Design Rationale & References

This page treats test selection as structured guidance rather than as an automatic verdict. That is why Licklider uses Welch's t-test as the default for independent two-group comparisons, treats normality checks as one input rather than as a hard switch, requires explicit confirmation of multiple- comparisons handling, and records manual overrides when the research design calls for a different choice.

1. Delacre, M., Lakens, D., & Leys, C. (2017). Why psychologists should by default use Welch's t-test instead of Student's t-test. International Review of Social Psychology, 30(1), 92-101. https://doi.org/10.5334/irsp.82
2. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108
3. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70.
4. Fisher, R. A. (1922). On the interpretation of chi-square from contingency tables, and the calculation of P. Journal of the Royal Statistical Society, 85(1), 87-94. https://doi.org/10.2307/2340521