How to check an AI assistant's statistical analysis: a 15-minute checklist

Quick answer. Before an analysis written or run by an AI assistant goes into your paper, run seven checks, which take about 15 minutes for a typical group comparison: (1) the unit counted as n is the animal or independent experiment; (2) the test matches the design you planned (paired or independent, how many groups, what kind of outcome); (3) the software defaults that change answers are set the way you intend — the defaults table below lists them with the one-line fix; (4) you re-run the code yourself with the versions printed; (5) you recompute one key number with a second tool; (6) every reference exists and says what is claimed; (7) the sentence you will publish matches the estimate and its interval. In the worked example below, the code ran without errors and three defaults still changed three conclusions.

Terms used below. Default: the setting a function uses when the code leaves it unset. Unit of analysis (n): the thing that was independently given a treatment, usually the animal. Spread: how variable a group’s values are, measured by its SD (equal spreads means equal variances). Welch’s t-test: compares two means without assuming equal spreads; the default this site recommends. Student’s t-test: requires equal spreads in both groups. Yates’ correction: an adjustment to the chi-square test for 2 × 2 tables that makes it more conservative. Adjusted p: a p-value corrected for running several comparisons. Estimate and interval: the size of the effect with its 95% confidence interval, the range of effect sizes compatible with the data. Degrees of freedom (df): a number every t or chi-square test prints; for two groups, Student’s test prints n₁ + n₂ − 2 and Welch’s prints a smaller decimal, estimated from the two spreads. SD: standard deviation, the typical distance of a value from its group mean. Expected count: what a cell of a 2 × 2 table would hold if group and outcome were unrelated (row total × column total ÷ grand total). Welch’s ANOVA: the version of one-way ANOVA (the standard comparison of three or more group means) that does not assume equal spreads. Kruskal–Wallis test: a test for three or more groups that compares the ranks (positions) of the values when all animals are sorted; Dunn’s test: the follow-up that compares every pair of groups on those ranks. Holm: a method that adjusts p-values for the number of comparisons. Seed: a fixed starting number that makes a random step give the same result every run. Bootstrap and Monte Carlo: methods that draw random samples (from your data, or simulated tables), so the result depends on the seed.

The checklist

Step	What to check	How	If it fails
1. Unit	n is animals or independent experiments, not cells, wells or images	Count the rows that went into the test and compare with the number of animals; check the df: for two groups, Student’s test prints n₁ + n₂ − 2 and Welch’s prints a smaller decimal. A df in the hundreds with 15 mice means cells were counted	Re-analyse per animal: what counts as n, nested data
2. Design → test	The test matches what you planned before seeing the data: paired or independent, two or more groups, measurement, score, count or yes/no	Name the test the code ran and compare it with your plan or the test chooser	Run the planned test; report any other analysis as additional
3. Defaults	No default from the table below silently changed the method	Search the code for each function name in the table	Add the fix from the table and re-run
4. Re-run	The output is reproducible on your machine, from your data	Run the code yourself with versions printed (the snippets below show how); before the test line, print the number of rows and the size of each group (len(df) and df.groupby("group").size() in Python, nrow(d) and table(d$group) in R) and compare with your animals. No code, only a summary from the chat? Ask for a complete script first (prompt 1)	Keep the versions with your notes; a result you cannot re-run — or an assistant that cannot produce code that runs — is not ready to report
5. Recompute	One key number (the test statistic or the difference) agrees with a second tool	Recompute it in R if the code was Python, or the other way round, or by the formula (for Welch’s t: the difference of means ÷ √(s₁²/n₁ + s₂²/n₂)); if it differs, check n first	Find which input or setting differs before trusting either
6. References	Each cited paper exists and contains the claim	Open the DOI (doi.org/…), then find the sentence in the paper	Remove the reference, or replace it with one you have read
7. Claim	The sentence matches the estimate, interval and test you report	Read the sentence next to the output	Rewrite it from the numbers: p = 0.06 or not significant, confidence intervals

Steps 1, 2 and 7 need your knowledge of the experiment; steps 3 to 6 are mechanical, and you can ask the assistant to draft them with the prompts below, then run the code and open the references yourself. For designs beyond a simple group comparison — repeated measures over time, survival, mixed models with several random effects — ask a statistician to review the analysis; the seven checks still apply but take longer than 15 minutes.

Software defaults that change the answer

Every default in the table below was read from the installed versions (SciPy 1.18.1, statsmodels 0.15.0, scikit-posthocs 0.17.1, Pingouin 0.7, seaborn 0.13.2, NumPy 2.5, R 4.3.3, rstatix 0.7.2, effectsize 0.8.6). These are defaults of the libraries, not of any assistant, and they can change between versions, so step 4 prints the versions. Use a fix only when it is the analysis you planned, whichever way the p-value then moves. The worked example below uses the SciPy ttest_ind, chi2_contingency and scikit-posthocs posthoc_dunn rows; read the rows whose function appears in your code, and if your function is not listed, use prompt 3 and set every argument explicitly.

Function	Default in the version read (versions above)	What it does to the answer	Fix	Details
SciPy stats.ttest_ind	equal_var=True (Student)	Assumes equal spreads; R’s t.test runs Welch by default, so the same data give different p-values	equal_var=False	Welch vs Student
statsmodels ttest_ind	usevar='pooled' (Student)	Same issue as SciPy’s default	usevar="unequal"	Welch vs Student
Pingouin ttest	correction='auto'	Welch only when the group sizes differ	correction=True	Welch vs Student
SciPy stats.f_oneway	equal_var=True (classical ANOVA)	Same issue for three or more groups	equal_var=False (SciPy 1.16+), or R oneway.test	Welch’s ANOVA
SciPy chi2_contingency, R chisq.test	Yates’ correction on for 2 × 2	More conservative p-value than the uncorrected test	When every expected count is 5 or more: correction=False / correct = FALSE; otherwise an exact test (Fisher’s, or Boschloo’s for more power)	Chi-square vs Fisher
SciPy boschloo_exact	Each column is one group	Passing groups as rows answers a different question	np.transpose(table) for a groups-as-rows table	Chi-square vs Fisher
SciPy fisher_exact on tables larger than 2 × 2	Monte Carlo estimate, no fixed seed	p changes slightly between runs	Set the random seed, or use R fisher.test	Chi-square vs Fisher
SciPy dunnett	rng=None	p changes slightly between runs	Set rng to a fixed seed	Which correction
statsmodels multipletests	method='hs' (Holm–Šidák)	R’s p.adjust defaults to Holm; the two differ slightly	method="holm", and name the method	Which correction
scikit-posthocs posthoc_dunn	p_adjust=None	Returns unadjusted p-values; R’s rstatix::dunn_test uses Holm	p_adjust="holm"	Kruskal–Wallis vs ANOVA
statsmodels MixedLM	z-based p-values	Too small with few animals	Per-animal means, or R lmerTest	Nested data
statsmodels ttost_ind	usevar='pooled'	Equivalence test assumes equal spreads	usevar="unequal"	p = 0.06 or not significant
R effectsize::omega_squared	alternative = "greater"	Prints a one-sided interval (upper limit 1)	alternative = "two.sided"	Reporting ANOVA
seaborn barplot, pointplot	errorbar=("ci", 95), bootstrap, no seed	Error bars are a bootstrap interval that covers less than 95% with a few animals	Pass the interval you intend, and say which in the legend	SD vs SEM
NumPy np.std	ddof=0	Population SD, smaller than the sample SD	ddof=1	SD vs SEM

Do this: search the code for every function name in the first column before you read any result.

A worked audit: three defaults, three changed conclusions

The data below are constructed and written by hand so that each default sits near a threshold. Imagine an assistant produced this Python and summarised it as “treatment raised the marker (p = 0.043); the outcome showed no difference in frequency (p = 0.059); high dose differed from low dose (p = 0.041)”. The code ran without errors. Here is the audit:

Analysis	As written (default)	After the check (fix)	Conclusion changes?
Marker, 5 treated vs 12 controls, spreads: SD 2.27 vs 0.36	Student: t(15) = 2.21, p = 0.043	Welch: t(4.08) = 1.40, p = 0.23 (the decimal df is Welch’s); difference 1.43 (95% CI −1.39 to 4.24)	Yes: no clear evidence of a rise (CI −1.39 to 4.24)
Outcome in 18 of 40 treated vs 9 of 40 controls (smallest expected count 13.5)	Yates: p = 0.059	No correction, as the chi-square guide recommends when every expected count is 5 or more: p = 0.033	Yes: more frequent with treatment
Histology grade 0–4, four doses, 6 animals each (Kruskal–Wallis H = 8.66, p = 0.034)	Dunn, unadjusted: low vs high p = 0.041	Dunn, Holm-adjusted: low vs high p = 0.21; vehicle vs high p = 0.039	Yes: only vehicle vs high differs

The second row shows that the check works in both directions: a default can hide a difference as well as create one. In every row the fix is the analysis you would have planned (step 2), whichever way the p-value then moves.

# Install once: pip install numpy pandas scipy scikit-posthocs
import platform
from importlib.metadata import version
import numpy as np, pandas as pd, scikit_posthocs as sp
from scipy import stats

print("Python", platform.python_version(), "| SciPy", version("scipy"), "| scikit-posthocs", version("scikit-posthocs"))

ctrl = np.array([10.2, 10.8, 11.1, 10.5, 10.9, 11.4, 10.7, 10.3, 11.0, 10.6, 11.2, 10.8])
trt = np.array([12.9, 10.1, 14.8, 9.6, 13.7])
print("Student (default):", stats.ttest_ind(trt, ctrl))
print("Welch (fixed):    ", stats.ttest_ind(trt, ctrl, equal_var=False))

tab = np.array([[18, 22], [9, 31]])       # rows: treated, control; columns: outcome yes, no
print("Yates (default):", stats.chi2_contingency(tab).pvalue)
print("No Yates (fixed):", stats.chi2_contingency(tab, correction=False).pvalue)

grades = dict(vehicle=[0, 1, 1, 1, 1, 2], low=[1, 1, 1, 1, 2, 2], mid=[1, 1, 2, 2, 2, 3], high=[1, 2, 2, 3, 3, 3])
df = pd.DataFrame([(g, v) for g, vs in grades.items() for v in vs], columns=["dose", "grade"])
print(stats.kruskal(*grades.values()))
print("Dunn, unadjusted (default):\n", sp.posthoc_dunn(df, val_col="grade", group_col="dose").round(4))
print("Dunn, Holm (fixed):\n", sp.posthoc_dunn(df, val_col="grade", group_col="dose", p_adjust="holm").round(4))

# Install once: install.packages("rstatix")
library(rstatix)   # dunn_test
sessionInfo()$R.version$version.string
packageVersion("rstatix")

ctrl <- c(10.2, 10.8, 11.1, 10.5, 10.9, 11.4, 10.7, 10.3, 11.0, 10.6, 11.2, 10.8)
trt  <- c(12.9, 10.1, 14.8, 9.6, 13.7)
t.test(trt, ctrl)                     # Welch: R's default
t.test(trt, ctrl, var.equal = TRUE)   # Student: what SciPy's default runs

tab <- matrix(c(18, 9, 22, 31), nrow = 2)   # rows: treated, control; columns: outcome yes, no
chisq.test(tab)$p.value                     # Yates: R's default too
chisq.test(tab, correct = FALSE)$p.value

d <- data.frame(dose = factor(rep(c("vehicle", "low", "mid", "high"), each = 6), levels = c("vehicle", "low", "mid", "high")),
                grade = c(0, 1, 1, 1, 1, 2,  1, 1, 1, 1, 2, 2,  1, 1, 2, 2, 2, 3,  1, 2, 2, 3, 3, 3))
kruskal.test(grade ~ dose, data = d)
dunn_test(d, grade ~ dose)   # Holm: rstatix's default

Both print the numbers in the audit table above — Student t = 2.21, p = 0.043; Welch t = 1.398 on 4.08 df, p = 0.23; Yates p = 0.059 and uncorrected p = 0.033; Kruskal–Wallis H = 8.66, p = 0.034; Dunn low vs high 0.041 unadjusted and 0.21 with Holm — so you can compare your run line by line. R prints the Welch test first because that is its default, and its output includes the difference’s 95% CI (−1.39 to 4.24); Python’s posthoc_dunn sorts the doses alphabetically, so its matrix rows come in a different order from R’s. The Welch t of 1.398 also follows from the formula: the difference of means ÷ √(s₁²/n₁ + s₂²/n₂).

What the published evaluations found

Numbers from studies of AI assistants age quickly, so each one is given with the model it tested and when; none of these studies evaluated newer models.

• In an evaluation of GPT-4 on univariate analyses of one clinical dataset (submitted for publication in June 2024), data processing and descriptive statistics were accurate in every trial, while on those univariate analyses GPT-4’s inferential results matched the expected values in 32.5% of trials with basic prompts, 81.3% with intermediate prompts and 92.5% with advanced prompts that specified the test and the steps (Ruta et al. 2025). The ChatGPT for statistical analysis page describes how that study scored the answers.
• On the StatQA benchmark of 11,623 statistical tasks, GPT-4o’s best score was 64.83%, and the models’ main error type was applying a method whose conditions were unmet (Zhu et al. 2024). That is what steps 1 and 2 look for.
• When ChatGPT was asked to write short literature reviews in 2023, the authors reported: “Within this set of documents, 55% of the GPT-3.5 citations but just 18% of the GPT-4 citations are fabricated. Likewise, 43% of the real (non-fabricated) GPT-3.5 citations but just 24% of the real GPT-4 citations include substantive citation errors.” (Walters & Wilder 2023). Step 6 opens every reference for this reason.

What to do: treat the numbers above as reasons for the checks rather than as a measure of the tool you use today; the checklist works the same whichever model wrote the code.

Prompts that make the checks faster

Keep identifiable participant data out of the chat; the ChatGPT for statistical analysis page covers uploads, and your institution’s rules decide what is allowed. Then paste these after the assistant has produced an analysis, and read its answers against your plan rather than accepting them.

1. “Give me the complete, runnable script you used, from reading the file to the final output, and list every statistical test you ran, the function and package for each, and every argument you left at its default value.”
2. “What is n in each test, and which column defines the experimental unit? Did you combine rows from the same animal?”
3. “Rewrite the code so that every default that affects the result is set explicitly, and print the software versions.”
4. “For each reference you cited, give the DOI and quote the sentence that supports the claim.” (Then open each DOI yourself, step 6.)
5. “Write the results sentence from the estimate and 95% confidence interval, without words such as ‘trend’ or ‘proves’.”

Keep a record

Sandve and colleagues’ first rule for reproducible computational research applies directly: “the critical details include the name and version of the program, as well as the exact parameters and inputs that were used” (Sandve et al. 2013). Save, with the data, the final code you ran, its printed versions and output, the second-tool recomputation from step 5 and the list of references you opened. Then disclose the assistant’s role as your journal asks: how to disclose an AI assistant’s role in a statistical analysis.

Do this: put the seven-step table in your lab notebook and tick off each check for every AI-assisted analysis before it reaches a draft.

References

• Ruta MR, Gaidici T, Irwin C, Lifshitz J (2025). ChatGPT for univariate statistics: validation of AI-assisted data analysis in healthcare research. Journal of Medical Internet Research 27:e63550. doi:10.2196/63550
• Zhu Y, Du S, Li B, Luo Y, Tang N (2024). Are large language models good statisticians? Advances in Neural Information Processing Systems 37. doi:10.52202/079017-2005
• Walters WH, Wilder EI (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports 13:14045. doi:10.1038/s41598-023-41032-5
• Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013). Ten simple rules for reproducible computational research. PLoS Computational Biology 9:e1003285. doi:10.1371/journal.pcbi.1003285