AB Testing and Applied Statistics
Hypotheses, sample size, statistical significance, p-value, Student's t-test, chi-square, interpretation
1What is a null hypothesis (H0) in an AB test?
What is a null hypothesis (H0) in an AB test?
Answer
The null hypothesis (H0) states that there is no significant difference between the two variants being tested. In AB testing, H0 asserts that any observed difference between the control group (A) and treatment group (B) is due to chance rather than the effect of the change being tested. The purpose of the statistical test is to determine whether the data allows us to reject H0.
2What is an alternative hypothesis (H1) in an AB test?
What is an alternative hypothesis (H1) in an AB test?
Answer
The alternative hypothesis (H1) is the opposite of the null hypothesis and asserts that there is a real difference between variants. In AB testing, H1 states that the change being tested has a measurable effect on the observed metric. If the statistical test allows us to reject H0 with sufficient confidence, we accept H1 as true.
3What is the p-value in an AB test?
What is the p-value in an AB test?
Answer
The p-value represents the probability of observing the obtained results (or more extreme) if the null hypothesis were true. The lower the p-value, the more unlikely it is that the results are due to chance. By convention, if the p-value is below the significance threshold (often 0.05), we reject H0 and consider the result statistically significant.
What is statistical significance in an AB test?
What is a Type I error (false positive) in an AB test?
+17 interview questions
Other Data Analytics interview topics
Google Sheets - Fundamentals
Google Sheets - Advanced Formulas
SQL - Fundamentals
SQL - Aggregations and Grouping
SQL - Joins
BigQuery - Fundamentals
Data Cleaning
KPIs and Business Metrics
Descriptive Statistics
Zapier and No-Code Automation
Data Visualization Principles
Python & Pandas - Fundamentals
Google Sheets - Automated Dashboards
SQL - Subqueries and CTEs
SQL - Window Functions
BigQuery - Advanced Features
Data Modeling
Funnel and Conversion Analysis
Cohort and Retention Analysis
Google Tag Manager and Tracking
APIs and Webhooks
dbt - Fundamentals
Looker Studio (Google Data Studio)
Power BI - Fundamentals
SQL - Advanced Analytical Queries
dbt - Advanced Features
Power BI - DAX and Advanced Dashboards
Python Analytics - Advanced Analysis and ML
Master Data Analytics for your next interview
Access all questions, flashcards, technical tests, code review exercises and interview simulators.
Start for free