Module 04B · Lesson 03

Methodology and Statistical Reasoning — Where Judgment Matters Most

Reading time: 22 minutes Track: Role Path — Computational Biology Prerequisites: Module 04B · Lessons 01 and 02

What this lesson does

The previous lesson covered code — the medium of your work. This lesson covers something harder: the methodology decisions that determine whether your work is correct.

This is where AI helps least and where domain training matters most. If you've been hoping AI would close the gap between you and a biostatistician — it won't. What AI does provide is a useful sounding board, faster access to method literature, and assistance with the mechanics of analysis. The judgment remains yours.

By the end of this lesson, you'll be able to:

Use AI to consult on methodology choices without being misled by confident wrong answers
Recognize the specific methodological pitfalls AI is most likely to gloss over
Identify when to use AI vs. when to bring in a biostatistician
Build defensible analytical workflows that survive review

This is the lesson where I push back hardest on naive AI use. Read carefully.

01 · What AI is genuinely good at in methodology

Let's start with the legitimate uses.

Use 1 — Orienting to unfamiliar methods

You encounter a method you don't know well. AI can:

Explain the conceptual basis
Identify when this method is appropriate
List its assumptions
Compare to related methods
Point you to canonical references

This is high-leverage orientation. You go from "I've heard of this" to "I understand what it does and when" in a focused conversation.

Use 2 — Understanding the assumptions of methods you use

Even for methods you use routinely, AI can surface assumptions you may take for granted:

What's the assumed distribution of the data?
What independence assumptions does this make?
When do these assumptions fail in practice?
How sensitive is the method to violations?

This is genuinely useful even for experienced analysts. The act of articulating assumptions often surfaces ones you'd internalized but not consciously noticed.

Use 3 — Method comparisons

For a given task, AI can produce structured comparisons:

Method A vs. Method B for [this kind of data]
Strengths and weaknesses of each
When you'd choose one over the other
Specific tools/packages that implement each

These comparisons accelerate decision-making. Verify against current method papers, but use as a starting point.

Use 4 — Power calculations and sample size reasoning

For standard setups, AI can:

Walk through power calculations
Identify the parameters that matter
Generate R/Python code for the calculations
Discuss the trade-offs

For non-standard setups, AI struggles. But for routine power analyses, it's quite useful.

Use 5 — Diagnostic interpretation

When you've run an analysis and see something diagnostic (Q-Q plot deviations, dispersion estimate weirdness, model fit issues), AI can:

Suggest what the diagnostic likely indicates
Recommend follow-up checks
Identify possible remediation

Again — useful for hypothesis generation, not for definitive interpretation.

02 · What AI is unreliable at

The other side. Be specific about where to be skeptical.

Unreliable 1 — Choosing the test for your specific situation

AI's recommendations for which statistical test to use are often wrong in subtle ways. The error patterns:

Suggesting parametric tests when non-parametric would be more appropriate (because parametric tests are more commonly mentioned in training data)
Defaulting to t-test or ANOVA when more sophisticated methods would be correct (mixed-effects models, longitudinal analyses)
Missing dependencies in your data (repeated measures, matched samples, clustering)
Generic recommendations that don't account for your sample size

The defense: Treat AI's test recommendation as a starting hypothesis. Verify by reading the relevant methods literature or consulting a biostatistician.

Unreliable 2 — Multiple comparisons reasoning

AI knows that "multiple comparisons require correction." It's less reliable about:

When correction is needed
Which correction is appropriate (Bonferroni vs. FDR vs. Holm vs. other)
The scope of correction (within an experiment vs. across experiments)
Whether certain comparisons are exploratory vs. confirmatory

The defense: Be deliberate about multiple comparisons in the analysis plan, before running the analysis. AI can help draft the plan; the plan itself should be reviewed by someone with statistical training.

Unreliable 3 — Effect size and biological significance

AI conflates statistical significance with biological meaningfulness. It will report "significant" findings without flagging that an effect size is tiny, or that a finding is statistically robust but biologically uninteresting.

The defense: Always report and interpret effect sizes, not just p-values. The interpretation is yours.

Unreliable 4 — Causal inference

AI is loose with causal language. It will use words like "causes," "drives," "leads to" when the underlying analysis only supports association.

The defense: Be precise about what your design can and can't support. For causal claims, you need designs that support causal inference (randomization, instrumental variables, careful natural experiments). Most -omics analyses are associational; AI may overstate.

Unreliable 5 — Cutting-edge methods

For methods introduced or revised in the past 12-18 months, AI's knowledge cutoff matters. New methods may be:

Not in its training at all (it'll either fabricate or punt)
In training but with outdated implementation details
In training but with shifted best practices

The defense: For recent methods, verify against current method papers and the tool's current documentation.

Unreliable 6 — Domain-specific quirks

Every -omics domain has quirks that aren't fully captured in general training data:

Single-cell data has specific normalization challenges
ChIP-seq has specific peak-calling considerations
Mass spec data has specific imputation considerations
Long-read sequencing has specific error-correction needs

The defense: AI is good for cross-domain orientation, less good for domain-specific best practices. Pair AI with domain-specific resources.

03 · The methodology consultation pattern

A specific workflow for using AI as a methodology consultant.

Pattern

Step 1 — State the problem precisely:

"Senior biostatistician with experience in [your domain]. I have a methodology question. The setup is:

Data: [specific description]

Sample size: [N per group, total]

Design: [factors, replication, structure]

Goal: [what you're trying to answer]

Constraints: [any limitations on methodology]

My current thinking: [your tentative approach]

Questions:

Is my current thinking sound? What would you push back on?

What alternative approaches should I consider?

What are the key assumptions and how should I check them?

What are the biggest risks in this analysis?"

Step 2 — Engage with the response:

The response is a starting point. Probe:

"You suggested X. Why X over Y?"
"If my sample size were smaller, would the recommendation change?"
"What if the data violated [assumption]?"

Step 3 — Verify externally:

Cross-check the recommendation against:

A methods paper for the suggested approach
Your biostatistician colleague (if available)
Comparable analyses in the published literature

Step 4 — Document the decision:

Once you've made a methodology choice, document:

What approach you chose
Why you chose it over alternatives
What assumptions you're making
How you verified them
What you'd do differently if assumptions failed

This document is part of your analysis. It survives the analysis and lets future you (or a reviewer) understand the choices.

04 · The biostatistician relationship

A specific point worth its own section.

Most biotechs have biostatistics support. As a computational biologist, you have a relationship with biostatisticians (in your company or as consultants). AI does not replace this relationship.

What biostatisticians do that AI doesn't

Take responsibility for the analysis being correct
Know your specific organization's standards and conventions
Have current knowledge of recent method developments
Can defend an analysis to regulators or reviewers
Understand the connection between specific analyses and downstream decisions
Can identify the right level of statistical sophistication for each context

Use AI to amplify the biostat relationship, not replace it

Better workflow with AI:

Use AI to come prepared to the biostatistician meeting — orient yourself to the methods, identify questions
Discuss methodology with the biostatistician
Use AI to implement the agreed methodology in code
Have the biostatistician review the results

Worse workflow with AI:

Use AI to choose the methodology
Implement and run without consultation
Hope the result holds up to review

The first workflow uses AI's strengths and biostatisticians' strengths complementarily. The second creates risk that compounds over time.

05 · Specific methodology areas — guidance

A few specific methodology domains where computational biologists frequently use AI, with calibration.

Domain: Differential expression analysis

AI is good for:

Pipeline boilerplate (count loading, model setup, contrast extraction)
Comparing DESeq2 vs. edgeR vs. limma
Visualization code
Pathway analysis follow-up

AI is weak on:

Choice of design when there are multiple factors
Handling batch effects appropriately
Interpretation of complex contrasts
Specific filtering thresholds for your data

Default with AI: Use for code; consult biostat for design.

Domain: Single-cell analysis

AI is good for:

Standard Scanpy/Seurat workflow boilerplate
Visualization
Standard QC steps
Differential expression within cell types

AI is weak on:

Quality control thresholds (these vary substantially by data type)
Choice of normalization and integration methods (this field shifts quickly)
Cell type annotation (AI can suggest but verification is essential)
Trajectory analysis interpretation

Default with AI: Use for code; verify thresholds and methodology against current single-cell best practices.

Domain: Variant calling and genomics

AI is good for:

Standard pipeline construction (BWA, GATK, etc.)
Variant annotation
Quality filtering boilerplate
Visualization

AI is weak on:

Caller choice for specific applications (somatic vs. germline, low-frequency variants, structural variants)
Filtering thresholds for your specific sequencing depth and quality
Handling tumor heterogeneity and clonal evolution
Clinical interpretation

Default with AI: Use for pipeline boilerplate; defer to domain specialists for clinical interpretation.

Domain: Machine learning

AI is good for:

Standard sklearn/PyTorch boilerplate
Model architecture suggestions for well-defined problems
Hyperparameter tuning patterns
Visualization

AI is weak on:

Choice of evaluation metric for the actual question
Handling imbalanced data appropriately
Preventing data leakage in cross-validation
Interpreting feature importance biologically

Default with AI: Use for implementation; bring statistical and ML training for design.

Domain: Protein structure and modeling

AI is good for:

Boilerplate for structure visualization
Standard analyses of structure data
Comparison of structural methods

AI is weak on:

Interpreting AlphaFold confidence metrics correctly
Understanding when comparative modeling is reliable
Connecting structure to function

Default with AI: Use for tools; verify interpretations.

06 · A worked example

A realistic scenario showing methodology work end-to-end.

Setting: You're analyzing data from a small (n=8) preclinical study. The endpoint is a continuous measure with high variability. The treatment has a hypothesized effect. You need to decide on the statistical approach.

Step 1 — Initial methodology consultation:

Senior biostatistician with experience in preclinical pharmacology. I have a study with:
- 8 animals total, 4 per group (treatment vs. control)
- Continuous endpoint (target engagement measured at one time point)
- High variability in similar studies (CV ~30-50%)
- Goal: determine if there's a treatment effect

My current thinking: t-test, with Welch's correction for unequal variances.

Questions:
1. Is t-test appropriate here, or should I consider non-parametric (Wilcoxon)?
2. What's the power with n=4 per group and CV ~40% to detect a 50% difference?
3. Should I be reporting effect size estimates and CIs rather than p-values?
4. Any other considerations for small-sample preclinical?

AI's response (summary):

Recommends Welch's t-test as reasonable default; suggests checking normality but notes that with n=4, normality is hard to assess and Wilcoxon may be more conservative
Power calculation: with the parameters given, power to detect a 50% difference is roughly 60-70%
Strongly recommends reporting effect size with CI rather than focusing on p-value, given small sample
Suggests considering a Bayesian approach for small samples (can incorporate prior knowledge); flags that this depends on whether you have justifiable priors

Step 2 — Probe deeper:

Walk me through how I'd implement the Bayesian alternative. What priors would be appropriate? How would I report results?

AI provides a structured response on Bayesian implementation with Stan or brms, suggests weakly informative priors, describes how to report posterior distributions.

Step 3 — Verify externally:

You check the recommendations against:

A methods paper on small-sample preclinical statistics
Your company's biostatistics SOP for preclinical
A quick consultation with a biostat colleague

The colleague says: t-test is fine for this context, but the effect-size-with-CI recommendation is important; the Bayesian alternative is overkill for this study but worth knowing about.

Step 4 — Implement:

Use AI to write the R code for Welch's t-test + effect size + CI + appropriate visualization.

Step 5 — Document:

Document the methodology choice and reasoning in the analysis report.

Total time: ~2 hours for methodology, code, and documentation. Without AI: probably 4-5 hours plus more consultation cycles.

The key: AI accelerated the orientation and code, but the methodology decision was verified externally before being committed to.

07 · The "stress test" before publication

A specific discipline worth building: before publishing or making decisions on the basis of an analysis, run a stress test.

The stress test

Step 1 — Reverse-engineer your conclusions:

Senior biostatistician reviewing the following analysis. I'm planning to conclude [your tentative conclusion]. Walk through:

1. Does the analysis actually support this conclusion?
2. What alternative conclusions are equally consistent with the data?
3. What's the weakest link in the analytical chain?
4. What would a skeptical reviewer push back on?
5. What additional analysis would strengthen or weaken the conclusion?

[Analysis details, key results, code summary]

Step 2 — Take the critique seriously:

The AI generates a list of potential issues. Some are real, some are reaches. Triage them:

For real issues: revise the analysis or weaken the conclusion
For reaches: note for your own thinking; don't necessarily address in writing

Step 3 — Sanity check the conclusion against your domain knowledge:

Does the conclusion match biological expectation? If yes, you may be confirming what you already believed; that's not necessarily bad but worth noting. If no, that's interesting — but also a reason to scrutinize the analysis harder.

This stress test takes 20-30 minutes. It catches problems before they become embarrassing.

08 · Knowledge check

Three questions.

Q1. Which of these is the most accurate characterization of AI's role in methodology choice for computational biology?

a) AI replaces the need for statistical training b) AI is a useful sounding board for orientation, method comparisons, and code implementation, but methodology decisions should be verified externally and AI is unreliable at choosing tests for specific situations c) AI is unreliable for any methodology work d) AI should only be used after a statistician approves the analysis

Q2. Why is AI unreliable at multiple comparisons reasoning specifically?

a) Multiple comparisons doesn't matter b) AI knows that "correction is needed" but is less reliable about when correction is needed, which method is appropriate, the appropriate scope of correction, and whether comparisons are exploratory vs. confirmatory c) AI doesn't know what multiple comparisons are d) Multiple comparisons is too advanced for AI

Q3. What's the recommended workflow for combining AI and biostatistician support?

a) Use AI to choose methodology, then have biostat review only if there's a problem b) Use AI for orientation and code; consult biostat for design decisions; use biostat review for important results — the workflows are complementary, not substitutive c) Don't use AI when biostat support is available d) Use biostat only for regulatory submissions; AI for everything else

Answers: Q1: b · Q2: b · Q3: b

09 · What's next

Lesson 04 of Module 04B: Communication and writing assistance — translating computational work for the rest of the organization.

End of Lesson 03.