p-Hacking Detection
p-hacking does not require intent. Exploring your data, trying different exclusions, testing a subgroup because it looked interesting - all of these are normal parts of scientific inquiry. The problem is when they are presented as confirmatory without disclosure. Licklider detects these patterns automatically and ensures they are transparently reported.
STEP 1 - The Pitfall: How p-Hacking Happens Without Knowing
The term "p-hacking" suggests deliberate manipulation, but the most common form is unintentional. A researcher generates a figure, sees a borderline result (p = 0.07), removes one outlier that "seemed suspect," and gets p = 0.04. They do not think of this as p-hacking - they think of it as "cleaning the data." The problem is that the exploratory exclusion-and-retest cycle has already inflated the effective Type I error rate.
The mathematics are unambiguous: with each additional analysis cycle on the same dataset, the probability of observing at least one false positive result increases substantially.
- 1 test, no iteration: alpha = 5% (nominal)
- 2 exclusion-and-retest cycles: effective alpha approximately 9.75%
- 4 cycles: effective alpha approximately 18.5%
- 6 cycles: effective alpha approximately 26.5%
- Subgroup exploration (5 subgroups): effective alpha approximately 22.6%
None of this is captured in the p-value that gets reported. The reviewer sees "p = 0.04" and has no way to know that it took six cycles of analysis to reach it.
STEP 2 - Journal Requirement
Science's reporting standards state that "all data collected, all experimental treatments, and all outcome measures" must be disclosed, including those that did not reach significance. Nature Human Behaviour requires a transparency statement describing the degree to which the analysis plan was pre-specified. The APA Publications Manual (7th ed.) requires authors to "report all manipulations, measures, and outcomes studied, even those that were not significant."
Pre-registration (on OSF, AsPredicted, or ClinicalTrials.gov) is the gold standard for confirmatory research, but it does not apply to exploratory analyses. For exploratory work, the current best practice is transparent disclosure of the analytic path: acknowledging the number of analyses tried, the decision points encountered, and the exploratory nature of the final result.
Licklider's p-Hacking Detection is designed to make this disclosure automatic - so that exploratory work is labeled as such, and confirmatory claims are made only when the analysis history supports them.
STEP 3 - Licklider's Solution
Input
- Analysis decisions made during the session - including data preparation steps, exclusion operations, and subgroup explorations - are monitored automatically. No additional setup is required.
- The Data Contract declaration of analysis intent: confirmatory, exploratory, or publication-ready (see Data Contract).
Output
- Session risk summary. A summary panel showing: number of distinct analysis variants run, number of exclusion-and-retest cycles, number of subgroups/filters explored, estimated effective alpha inflation.
- Risk level classification. Sessions are classified as Low / Moderate / High risk based on the number and pattern of iterative decisions made. The classification is shown on the export panel and included in the transparency disclosure block.
- Transparency disclosure block. A pre-written transparency statement (see STEP 4) that is automatically included in the export package when risk is Moderate or High.
- Session timeline access. A "View full session timeline" link exposes the chronological list of relevant analysis events behind the risk summary.
Guard
Guard condition - High risk, confirmatory intent declared: If the Data Contract declares the analysis as confirmatory or publication-ready and the session risk level reaches High, Licklider blocks export and requires the researcher to either (a) re-declare the analysis as exploratory, or (b) acknowledge the transparency disclosure and add it to the Methods section.
Guard condition - Moderate risk: Export proceeds, but the transparency disclosure block is automatically inserted into the generated Methods text. The researcher cannot remove it without manually editing the export.
STEP 4 - Ready-to-Publish Output
Transparency disclosure - Exploratory analysis (Moderate risk)
The analysis reported here was exploratory in nature. During the analysis
session, four analysis variants were examined, including two exclusion-and-
retest cycles. These iterations were conducted to understand the sensitivity
of the result to data preparation choices and were logged automatically by
Licklider (risk level: Moderate). The
reported result reflects the analysis most consistent with the pre-specified
outcome measure. Readers should interpret the reported p-value with awareness
that it reflects one of several analyses conducted on the same dataset.Transparency disclosure - Low risk (confirmatory)
The primary analysis was pre-specified in the Data Contract prior to data
collection. No exclusion criteria were modified after initial specification,
and no subgroup or filter exploration was performed during the analysis
session (risk level: Low). The reported
result represents the single pre-specified primary analysis.