Statistical Analysis and Experimental Design

• Design A/B tests with proper experimental methodology: define the metric, set minimum detectable effect (MDE), choose significance threshold (alpha), and calculate required sample size using power analysis Create
• Conduct hypothesis tests appropriate to data type: chi-squared tests for proportions, t-tests for continuous metrics, Fisher's exact test for small samples, and Mann-Whitney U for non-normal distributions Apply
• Calculate and interpret confidence intervals for differences between groups, communicating both statistical significance and practical significance to stakeholders Apply
• Detect common statistical pitfalls in real analyses: multiple testing problems, novelty effects, Simpson's Paradox, survivorship bias, and p-hacking Evaluate
• Build logistic regression models in Python (scikit-learn/statsmodels) to predict binary outcomes, checking for multicollinearity (VIF), interpreting coefficients, and validating with holdout data Apply
• Analyze cohort retention curves and segment-level performance differences, identifying whether aggregate results mask meaningful subgroup variation Analyze
• Communicate uncertainty to non-technical stakeholders using language they trust: ranges instead of point estimates, confidence levels instead of p-values, and explicit caveats on what the data does and does not tell them Apply
• Evaluate when a rules-based heuristic outperforms a machine learning model given available data volume, feature quality, and interpretability requirements Evaluate

Levels: Remember · Understand · Apply · Analyze · Evaluate · Create — highest demands most original thinking.