A/B Test Results: Ad vs. PSA Conversions
Problem Statement
The marketing team wanted to better understand the effectiveness of paid Ads versus PSAs but lacked clear visibility into conversion performance. Specifically, they needed to identify:
Which campaign type (Ad vs. PSA) delivers higher conversions
What times of day and days of the week drive the strongest results
How many times an ad should be shown to a customer before conversions start to drop (optimal ad frequency)
Without these insights, the business risked inefficient budget allocation, overspending on low-performing campaigns, and missed opportunities to maximize ROI and customer engagement.
Ask
How can we understand our coffee shop’s sales performance to identify best-selling products, peak sales hours, and customer payment preferences? By answering this, the business can optimize inventory, adjust staffing schedules, and design promotions that align with customer behavior
Prepare
Marketing A/B testing dataset
Marketing companies want to run successful campaigns, but the market is complex and several options can work. So normally they tun A/B tests, that is a randomized experimentation process wherein two or more versions of a variable (web page, page element, banner, etc.) are shown to different segments of people at the same time to determine which version leaves the maximum impact and drive business metrics.
Data set from Kaggle
🔗:https://www.kaggle.com/datasets/faviovaz/marketing-ab-testing
Process
Cleaned the data in Excel:
Cleaned the data in Excel:
Removed duplicates and blank entries
Standardized date/time format
Grouped Hour into ranges (Morning, Afternoon, Evening, Night).
Normalized frequency bins (0–10, 11–50, 51–100, etc.).
Created calculated fields:
Day type
Hour Label
AD conversion rate
Lift %
PSA conversion rate
Analyze
Total Conversions: 14,843
Lift %: Ads outperform PSA by 43%.
Conversion Rates: Ad = 2.55%, PSA = 1.79%.
Conversion peaks when users see ads 51–100 times, then declines (diminishing returns).
Conversion highest in late afternoon & evening (3–8 PM), peaking above 3%.
Morning & night hours see lower conversion rates (<2.5%).
Monday & Tuesday afternoons (3.81% & 3.56%) show the strongest results.
Weekends (Saturday & Sunday) underperform.
Act
Campaign Strategy 🎯
Prioritize Ads over PSA, given their 43% higher conversion lift.
Scale ad budgets for Monday–Tuesday afternoons and weekday evenings.
Scheduling & Targeting ⏰
Optimize delivery windows: 3 PM – 8 PM for maximum impact.
Reduce spend during mornings and weekends (low conversion performance).
Ad Frequency 📊
Cap exposure around 51–100 impressions per user.
Avoid overspending on high-frequency exposures (100+), which reduce efficiency.
Optimization 🚀
Continuously monitor lift % as campaigns scale.
Test creative variations during peak time slots to further boost conversions.
Reevaluate PSA messaging or reduce spend, reallocating towards high-performing Ads.