by Marco Elumba
In a strategic initiative to enhance player engagement and increase revenue, an A/B test was conducted targeting new users of our game. The test focused on the potential benefits of offering an additional restart opportunity by watching a rewarded ad after a loss. This initiative aimed to determine whether providing an extra ad-driven restart could improve retention and boost revenue.
The results of the A/B test suggest that offering an additional ad for a second restart does not significantly impact revenue or player retention. This finding implies that our current strategy of one ad per loss may be optimised for our game's monetisation and user experience balance. The lack of statistical significance supports maintaining the current ad model without additional adjustments.
The A/B test divided new users into two groups over a 15-day period. The control group had the option to watch one ad for an extra life after their first loss, while the test group could watch ads for extra lives after their first two losses. Analysis of the revenue impact used a Frequentist approach via the Mann-Whitney U test, resulting in a T-statistic of 4490443082.500 and a P-value of 0.329, indicating no statistically significant difference in revenue generation between the two groups.
import pandas as pd
from scipy import stats
import matplotlib.pyplot as plt
import seaborn as sns
# Load your data
data = pd.read_csv('cohort_data.csv')
# Ensure that 'ad_revenue' is treated as a float if it's not
data['ad_revenue'] = data['ad_revenue'].astype(float)
# Split data into control and test groups based on 'ab_group'
control_group = data[data['ab_group'] == 'control_group']['ad_revenue']
test_group = data[data['ab_group'] == 'test_group']['ad_revenue']
# Perform a t-test
t_stat, p_value = stats.mannwhitneyu(control_group, test_group)
# Printing the results
print(f"T-statistic: {t_stat:.3f}")
print(f"P-value: {p_value:.3f}")
if p_value < 0.05:
print("There is a statistically significant difference in average revenue between the control and test groups.")
else:
print("There is no statistically significant difference in average revenue between the control and test groups.")
# Histograms of both groups
plt.figure(figsize=(12, 6))
sns.histplot(control_group, color="skyblue", kde=True, label='Control Group')
sns.histplot(test_group, color="red", kde=True, label='Test Group')
plt.legend()
plt.title('Distribution of Ad Revenue: Control vs. Test Group')
plt.xlabel('Ad Revenue')
plt.ylabel('Frequency')
plt.show()
# Box plot for both groups
plt.figure(figsize=(8, 6))
sns.boxplot(x='ab_group', y='ad_revenue', data=data)
plt.title('Box Plot of Ad Revenue: Control vs. Test Group')
plt.show()
Further Bayesian analysis confirmed these findings with mean revenue probabilities practically equal between the groups and a mere 3% likelihood that the test group's revenue was higher.
import pymc as pm
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import arviz as az
# Load your data
data = pd.read_csv('cohort_data.csv')
# Ensure that 'ad_revenue' is treated as a float if it's not
data['ad_revenue'] = data['ad_revenue'].astype(float)
# Split data into control and test groups based on 'ab_group'
control_data = data[data['ab_group'] == 'control_group']['ad_revenue']
test_data = data[data['ab_group'] == 'test_group']['ad_revenue']
# Bayesian A/B Testing Model
with pm.Model() as model:
# Priors for unknown model parameters
p_control = pm.Normal('p_control',
mu=control_data.mean(),
sigma=control_data.std())
p_test = pm.Normal('p_test', mu=test_data.mean(), sigma=test_data.std())
# Likelihood (sampling distribution) of observations
control_obs = pm.Normal('control_obs',
mu=p_control,
sigma=control_data.std(),
observed=control_data)
test_obs = pm.Normal('test_obs',
mu=p_test,
sigma=test_data.std(),
observed=test_data)
# Define the deterministic delta function. This is our parameter of interest.
delta = pm.Deterministic('delta', p_test - p_control)
# Drawing 3000 posterior samples
trace = pm.sample(3000, return_inferencedata=True)
# Plotting the posterior distribution of the parameter of interest
az.plot_posterior(trace, var_names=['p_control', 'p_test', 'delta'], ref_val=0)
plt.show()
# Results
delta_samples = trace.posterior['delta'].values.flatten()
probability_test_better = np.mean(delta_samples > 0)
print(f"Probability that test group has higher revenue than control: {probability_test_better:.2f}")