Analytics Methods Cheat Sheet — Decision-focused Recipes

Purpose

This compact cheat sheet gives analysts fast, repeatable analysis recipes tied to specific learning questions and decisions. Each method entry explains what the method does, when to choose it, the data you need, a short implementation checklist, validation steps to guard against common errors, and guidance for interpreting results so findings reliably inform experiments and decisions.

Before you begin (cross-method checklist)

• Define the decision or learning question: What action would you take on different outcomes? State the null and alternative in practical terms.
• Owner & context: Assign an analysis owner, tag related dashboards, experiments, and stakeholders.
• Metric definitions: One clear primary metric (and definitions for secondary metrics). Record calculation SQL/formula and segmentation rules.
• Data snapshot & lineage: Note sources, extraction time, quality checks, and any transformations applied.
• Sample size & power: Roughly ensure you have enough data to detect the minimum practical effect for your decision.
• Pre-specify analysis plan: When possible, pre-register hypotheses, primary comparisons, and stopping rules to avoid p-hacking.
• Reproducibility: Save code/notebooks, parameter values, and a brief README describing steps to reproduce.

Methods

Cohort Analysis

What: Track groups of users/events defined by a shared starting point (signup date, first purchase) over time.

When to use: Understand retention, lifetime value, onboarding effects, or changes across vintages.

Required data: Event timestamps, user identifiers, relevant event types or value metrics.

Quick checklist:

• Define cohort key and time buckets (days/weeks/months).
• Consistent metric calculations across cohorts.
• Normalize for cohort size (use rates or per-user averages).

Validation: Check cohort size stability, inspection for data gaps, and confirm no changing definitions across cohorts.

Interpretation: Look for shifts in shape (decay rate) and level; test whether observed differences exceed expected sampling variability.

Anomaly Detection

What: Identify points or intervals where metrics deviate unexpectedly from baseline.

When to use: Monitoring, incident triage, detecting regression after releases, or QA on pipelines.

Required data: High-frequency metric time series, seasonal/context signals (day-of-week, campaign flags).

Quick checklist:

• Choose baseline window and seasonality model.
• Set alert thresholds that balance false positives and misses.
• Annotate known events/releases to reduce false alarms.

Validation: Backtest on historical incidents; inspect residuals for nonstationarity.

Interpretation: An anomaly signals a change needing investigation—don’t jump to causal claims without further analysis.

Causal Inference / Randomized (A/B) Test

What: Randomized assignment to measure causal effect of a treatment on outcomes.

When to use: When you can randomize exposure and want an unbiased estimate of effect for a decision.

Required data: Randomization assignment, outcome measures, exposure logs, pre-period covariates.

Quick checklist:

• Confirm randomization mechanism and treatment assignment logs.
• Pre-specify primary metric and analysis window.
• Check balance on covariates and run power calculation before starting (or plan an adaptive design).

Validation: Test for balance, check for logging gaps, run sensitivity checks (intent-to-treat vs. per-protocol).

Interpretation: Use confidence intervals and minimal detectable effect—assess practical significance and risk of adverse side effects.

Regression Analysis (Linear / Logistic)

What: Model relationships between predictors and an outcome, controlling for covariates.

When to use: Estimate associations, adjust for confounders, or build predictive baselines—use for hypothesis generation or adjusted comparisons when randomization is impossible.

Required data: Outcome, candidate predictors, covariates, sufficient sample size, and diagnostic variables.

Quick checklist:

• Specify model form and test functional form (transformations, interactions).
• Check multicollinearity and influential points.
• Split data for validation when predictive power matters.

Validation: Residual diagnostics, out-of-sample performance, and robustness to alternative specifications.

Interpretation: Coefficients show adjusted associations—not definitive causal effects without additional identification arguments.

Funnel / Conversion Analysis

What: Measure conversion rates across ordered steps in a user flow.

When to use: Diagnose where users drop out and prioritize flow improvements or experiments.

Required data: Time-stamped events for each funnel step and unique identifiers.

Quick checklist:

• Define strict step membership rules and time windows between steps.
• Segment by relevant cohorts (device, geography, campaign).

Validation: Confirm event instrumentation, dedupe users, and check sessionization logic.

Interpretation: Prioritize steps with large absolute drop or high-value user impact; measure lift with experiments if possible.

Time Series & Forecasting

What: Model temporal patterns for forecasting or understanding trend/seasonality.

When to use: Demand planning, capacity, or any metric with meaningful temporal structure.

Required data: Clean, regular-interval time series; external regressors if relevant (promotions, holidays).

Quick checklist:

• Decompose into trend, seasonal, and residual components.
• Validate on holdout periods; quantify forecast uncertainty.

Validation: Backtest, check residual autocorrelation, and compare simple baselines (naïve or seasonal naïve).

Segmentation & Clustering

What: Group observations into meaningful segments for targeting or analysis.

When to use: Personalization, cohort discovery, or to simplify complex populations.

Required data: Representative feature set (behavioral, demographic, transactional).

Quick checklist:

• Standardize features, pick an interpretable number of segments, and validate stability.
• Profile segments with business-relevant summaries.

Validation: Check segment robustness across random seeds and time periods; ensure segments map to actionable differences.

Diagnostic / Root-Cause Drilldown

What: Systematic drilldown combining segmentation, time series, and event inspection to explain a change.

When to use: After detecting a meaningful anomaly or when a key metric shifts unexpectedly.

Required data: Cross-cutting dimensions (product, region, channel), logs, release and campaign annotations.

Quick checklist:

• Compare delta decomposition across dimensions; inspect top contributors by absolute and relative change.
• Triangulate with instrumentation, deployments, and external events.

Validation: Confirm candidates with additional queries, replicate on raw data, and, if possible, test corrective hypotheses experimentally.

Common cross-method pitfalls

• Treating correlation as causation without identification strategy.
• Changing metric or cohort definitions mid-analysis.
• Underpowered tests or multiple uncorrected comparisons.
• Poor instrumentation or silent logging gaps.
• Ignoring business context—statistical significance alone is not a decision rule.

Quick links & templates

Use these starter resources to operationalize findings:

• Experiment template (pre-specification checklist)
• Dashboard spec template (metric definitions & annotations)
• Analysis reproducibility README (notebook + data snapshot checklist)

If you want a tailored checklist for a specific method or to capture analysis metadata automatically, consider converting this cheat sheet into an interactive analysis intake form that saves the plan and links to experiment/dashboards.

Recommended next steps

1. Pick the method matching your decision question and follow the cross-method checklist first.
2. Save your analysis plan and artifacts with an owner and date.
3. If the result will trigger action, predefine acceptance criteria and a rollout plan (pilot → scale → monitor).