Calcium imaging analysis: ΔF/F, neurons within animals and selecting responsive cells

Quick answer. Neurons imaged in one mouse are not independent, so compare groups with the mouse as n (one mean per mouse) or a mixed model with mouse as a random effect and t-based tests (z-tests are too liberal with few mice). With neurons as n, 57.2% of our simulated studies found a “significant” genotype difference that did not exist; with mice as n, 4.2%, though with 5 mice per group the mouse-level test caught a large real drop in only 25.1%, so plan enough mice (how many biological replicates). If you select “responsive” neurons, choose them on different trials from those you use to measure their response (for example odd vs even trials): in our simulation, measured on the same trials, the selected neurons’ responses were inflated by about 25% (by 9% in the worked example, 0.143 against 0.131). Set the ΔF/F baseline window and the responsiveness rule before looking at the results.

Terms used below. ΔF/F: the change in fluorescence relative to baseline, (F − F0) ÷ F0. Baseline (F0): the fluorescence before the stimulus, from a window set in advance. Response: here, the mean ΔF/F in the response window of a trial. Trial: one presentation of the stimulus. Responsive neuron: a neuron whose responses pass a rule, here a one-sample t-test across trials with p < 0.05 and a positive mean; with no correction for testing hundreds of neurons, about 2.5% of truly unresponsive neurons pass by chance (the 5% two-sided rate, halved by requiring a positive mean). Double dipping (circular analysis): using the same data to select cells and to measure them. Held-out trials: trials not used for selection. Session: one imaging run, often on a separate day. Motion correction: aligning frames to remove brain movement. Neuropil: the diffuse fluorescence around a cell body, often partly subtracted. Percentage points: the difference between two percentages (47% − 38% = 9 percentage points). Mixed model: a regression with a random effect for each mouse, so neurons from one mouse are not treated as independent. Welch’s t-test: compares two groups without assuming equal spreads. z-test vs t-based (Satterthwaite) test: ways a mixed model computes p-values; z-tests ignore how few mice there are and give too many false positives, t-based tests do not. 95% confidence interval (CI): the range of values compatible with the data. Power: the chance of detecting an effect that is really there. False positive: a “significant” result when there is no real effect.

Which analysis for your question

Question	Analysis	Report
Does the average response differ between groups?	One mean per mouse, Welch’s t-test (mouse as n); or a mixed model with mouse as a random effect and t-based (Satterthwaite) tests, since z-tests are too liberal with few mice	Difference with 95% CI, n mice and neurons per mouse
Does the share of responsive neurons differ?	Share per mouse, Welch’s t-test with mice as n (a binomial mixed model’s z-test is too liberal with a handful of mice)	Difference in percentage points with 95% CI
How strongly do responsive neurons respond?	Select on some trials, measure on held-out trials; to compare groups, average the held-out responses per mouse	Mean held-out response with 95% CI (mice as n), selection rule
Same neurons before and after a drug	Change per neuron, averaged per mouse, then a one-sample or paired test across mice	Change with 95% CI (mice as n)
Many neurons, few mice	Plan more mice; more neurons per mouse adds little	—

Aarts and colleagues found that 53% of 314 neuroscience papers included designs “in which multiple observations are collected from a single research object (for example, multiple neurons from one animal)”, and that ignoring this gives false-positive rates “up to 80%”; multilevel (mixed) models or analysis of one value per animal fix it (Aarts et al. 2014). The ARRIVE guidelines explain that “measurements from 50 individual cells from a single mouse represent n = 1 when the experimental unit is the mouse” (Percie du Sert et al. 2020). On selection, Kriegeskorte and colleagues warn that double dipping “will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis”, and give “the restriction of in-depth analysis to neurons with certain response properties” as an example (Kriegeskorte et al. 2009). For the baseline, a methods primer notes that “baseline fluorescence values should be stable throughout the duration of recording”, since drift can indicate “photobleaching due to excessive laser intensity or focus drift” (Grienberger et al. 2022).

Worked example: WT and KO mice with the same true response

Constructed data: stimulus responses (ΔF/F) of 40 neurons in each of 5 wild-type (WT) and 5 knockout (KO) mice, 20 trials per neuron. By design, both genotypes have the same average response, ΔF/F 0.05. Each mouse has its own average (a standard deviation of 0.04 between mice), each neuron its own response around it (0.06), and trials scatter around each neuron’s response (0.15). The random seed was the first one tried and was not chosen for its result. Stored code and data: responses.csv, model.py, make_data.py, analysis.py, analysis.R and the analysis output; analysis.py and analysis.R print the same output.

Analysis	Result
Neurons as n (200 vs 200), Welch’s t-test	KO − WT −0.022, p = 0.0031 (“KO responds less”)
Mice as n (5 vs 5), Welch’s t-test	KO − WT −0.022 (95% CI −0.061 to 0.017), p = 0.22
Classified responsive using all 20 trials	170 of 400 (42.5%); their mean ΔF/F on those same trials 0.122 (inflated), against 0.054 for all neurons
Selected on odd trials (103 neurons)	Mean ΔF/F 0.143 on the odd trials, 0.131 on the held-out even trials (pooled across mice, descriptive)
Share of responsive neurons per mouse	WT 47.0%, KO 38.0%; mice as n, p = 0.39

With neurons as n, the small difference between these two groups of mice looks certain; with mice as n, the interval runs from a clear decrease to a small increase. The responsive neurons’ average (0.122) describes those neurons only, not the population (0.054), and measuring them on the trials that selected them adds a further upward bias. In 10,000 simulated studies of this design (simulation.py, the simulation output):

Scenario	Neurons as n, p < 0.05	Mice as n, p < 0.05	Responsive neurons (selected on odd trials): mean ΔF/F on the same trials	Mean ΔF/F on held-out trials
No genotype difference	57.2% (false positives)	4.2% (false positives)	0.150	0.119
KO lower by 0.04 (0.05 vs 0.01)	84.7%	25.1% (power)	0.146	0.112

Neurons as n gives false positives in more than half of the studies with no difference; mice as n keeps them near 5% but, with 5 mice per group, detects a large drop in only 25.1% of studies. Measured on their selection trials, responsive neurons look about 0.03 (25%) stronger than on held-out trials. If an analysis gives one p-value for the mean response and another for the share of responsive neurons, report both at the mouse level rather than choosing the more striking one.

The code expects one row per trial with columns mouse, genotype (WT or KO), neuron, trial and dff, the response for that trial: (mean F in the response window − mean F in the baseline window) ÷ mean F in the baseline window, with both windows fixed in advance; rename your columns and labels to match, and run the code in the folder that holds the CSV file. Python needs NumPy, pandas and SciPy; the R code uses base R only.

# Condensed: analysis.py also gives the neurons-as-n test and the share of responsive neurons
import pandas as pd
from scipy import stats

d = pd.read_csv("responses.csv")
m = d.groupby(["mouse", "genotype"]).dff.mean().reset_index()          # one mean per mouse
t = stats.ttest_ind(m.dff[m.genotype == "KO"], m.dff[m.genotype == "WT"], equal_var=False)
print(t, t.confidence_interval())                                        # KO - WT, mice as n: p = 0.22, CI -0.061 to 0.017
odd, even = d[d.trial % 2 == 1], d[d.trial % 2 == 0]                     # select on odd, measure on even trials
g = odd.groupby(["mouse", "neuron"]).dff
sel = g.apply(lambda v: stats.ttest_1samp(v, 0).pvalue < 0.05 and v.mean() > 0)
h = even.groupby(["mouse", "neuron"]).dff.mean()[sel]                    # held-out response of responsive neurons
print(h.mean(), h.groupby(level="mouse").mean())                         # pooled 0.131; per mouse, for group comparisons

d <- read.csv("responses.csv")
m <- aggregate(dff ~ mouse + genotype, data = d, FUN = mean)             # one mean per mouse
t.test(m$dff[m$genotype == "KO"], m$dff[m$genotype == "WT"])             # KO - WT, mice as n
odd <- d[d$trial %% 2 == 1, ]; even <- d[d$trial %% 2 == 0, ]
s <- aggregate(dff ~ mouse + neuron, data = odd, FUN = function(v) t.test(v)$p.value < 0.05 && mean(v) > 0)
e <- aggregate(dff ~ mouse + neuron, data = even, FUN = mean)
mean(e$dff[s$dff == 1])                                                  # held-out response of responsive neurons

Before you analyse

1. Fix the preprocessing: motion correction, the baseline window for F0, neuropil handling and the response window, before comparing groups; CaImAn, for example, detrends traces “to remove non-stationary effects, for example photo-bleaching” (Giovannucci et al. 2019).
2. Plan the number of mice, not neurons; report both, and the neurons per mouse (what counts as n).
3. Write the responsiveness rule in advance (test, threshold, trials) and keep selection and measurement on separate trials or sessions; with fewer than about 10 trials per neuron, select in one session and measure in another (add a session column and keep neuron IDs matched across sessions). If the selection was already made on all trials, report the selected neurons’ responses as descriptive and say they are biased upwards.
4. Compare shares of responsive neurons per mouse, not pooled across mice (comparing percentages).
5. Image the groups in the same way: the same indicator, expression time, laser power and depth, interleaved across days, so that brightness differences are not mistaken for genotype differences.

Check what an AI assistant ran

1. The assistant compared hundreds of neurons between groups. Ask for one value per mouse, or a mixed model with mouse as a random effect.
2. The assistant selected responsive neurons and then tested their responses on the same trials. Ask for selection on separate trials.
3. The assistant pooled all neurons to compare the percentage of responsive cells. Ask for the percentage per mouse.
4. The assistant changed the baseline window or responsiveness threshold after seeing the results. Ask for the rule set in advance and the results under it.

Report it

ΔF/F was computed as (F − F0)/F0 with F0 the mean fluorescence [1 s] before each stimulus; the response was the mean ΔF/F over the [2 s] after stimulus onset. Genotypes were compared with one mean per mouse (Welch’s t-test; [5] mice per genotype, [40] neurons per mouse). Neurons were classed as responsive on odd trials (one-sample t-test, p < 0.05, positive mean), and their response was measured on even trials (R [4.3.3]).

The mean response did not differ clearly between genotypes (KO − WT [−0.022] ΔF/F, 95% CI [−0.061] to [0.017]; p = [0.22]). [25.8]% of neurons were responsive on odd trials; their response on even trials was [0.131] ΔF/F. The share of neurons classed responsive per mouse using all 20 trials, a less strict rule than odd trials alone, was [47.0]% in WT and [38.0]% in KO mice (p = [0.39], mice as n).

Replace the bracketed values with your own.

References

• Aarts E, Verhage M, Veenvliet JV, Dolan CV, van der Sluis S (2014). A solution to dependency: using multilevel analysis to accommodate nested data. Nature Neuroscience 17:491–496. doi:10.1038/nn.3648
• Percie du Sert N, Ahluwalia A, Alam S, et al. (2020). Reporting animal research: explanation and elaboration for the ARRIVE guidelines 2.0. PLOS Biology 18:e3000411. doi:10.1371/journal.pbio.3000411
• Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nature Neuroscience 12:535–540. doi:10.1038/nn.2303
• Grienberger C, Giovannucci A, Zeiger W, Portera-Cailliau C (2022). Two-photon calcium imaging of neuronal activity. Nature Reviews Methods Primers 2:67. doi:10.1038/s43586-022-00147-1
• Giovannucci A, Friedrich J, Gunn P, et al. (2019). CaImAn an open source tool for scalable calcium imaging data analysis. eLife 8:e38173. doi:10.7554/eLife.38173