This project aims to compare the performance of current (Max-bidding) and new campaign (Average-bidding) for an e-commerce website to make a decision about which campaign to use in the future based on A/B test results by EDA and hypothesis testing.

To get further information about exploratory data analysis and hypothesis testing, please check out the detailed report.

The datasets named as Control campaign (Max-bidding) and Test campaign (Average-bidding) contains 30 rows and 10 attributes. Each row corresponds to A/B test results of the company for Control and Test campaigns on August 2019. The attributes are as follows:

• Campaign Name: Target campaign type for ad landing page.

• Spend [USD]: The amount of money spent on advertising in the campaign.

• # of Impressions: The number of people who viewed the ad in the campaign (contains repeated viewing of the same person for the ad).

• Reach: The number of unique people who saw the ad in the campaign.

• # of Website Clicks: The number of users who clicked on the website link in the campaign's advertisement.

• # of Searches: The number of users who performed a search on the website.

• # of View Content: Number of users who have viewed product details.

• # of Add to Cart: The number of users who have added the product to the cart.

• # of Purchase: The number of users who have purchased the product.

The link to the datasets is here.

1) Missing Values Handling:

7 out of 10 columns excluding "campaign_name","date" and "spend_usd" have missing values in only one row. To handle this, missing values are replaced with mean values of their own column.

2) Duplicated Values Handling:

A/B Test Results datasets which are control campaign and test campaign do not have any duplicated records.

3) Changing Data Types:

The data type of "Date" column format is corrected from "object" to "datetime64[ns]".Additionally, the columns whose data types are "float64" are changed to "int64" to get the same format in both datasets.

4) Adding Performance Metrics:

To gauge which campaign are most appealing to the users, new performance metrics are identified :

• CTR(%) = Website clicks/ Impressions * 100
• Conversion Rate(%) = Purchase / Website clicks * 100

1) Descriptive Statistics

Before running hypothesis testing, to provide a concise summary of the data, the measures of central tendency(mean, median), variability(standard deviation, variance, IQR, and percentiles), skewness, and distribution as well as outliers are analyzed by using box plots.

2) What is the total number of records for each attribute per campaign ?

To gain a general overview about campaigns, pie charts are used to compare the total values of each attribute.

3) What is the relationship between attributes ?

Scatter plots are taken into consideration to get an idea about the strength and direction of the relationship between attributes.

4) What is the comparison of CTR(%) and Conversion Rate(%) per campaign ?

Time-series plots are included to see the performances of both campaigns over time and determine any trends or pattern in the performance of CTR(%) and Conversion Rate(%).

Hypothesis Testing is considered to determine whether the difference of CTR(%) and Conversion Rate(%)in performance between Control campaign and Test campaign is statistically significant or not. From the perspective of assumptions of statistical test such as normality, homogeneity of variances, random sampling, sample size and data type, tests are selected to make decisions about the campaigns as follows:

1) Shapiro-Wilk Test

It is performed to check the normality of campaigns in CTR(%) and Conversion Rate(%).

2) Fligner-Killeen Test

It is performed to check the homogeneity of variances of campaigns in CTR(%) and Conversion Rate(%).

3) Mann-Whitney U test

Based on the results of Shapiro-Wilk Test and Fligner-Killeen Test, Mann-Whitney U test is selected as a non-parametric test to find whether the medians of campaigns in CTR(%) and Conversion Rate(%) are equal or not.

• Test campaign, which is Average-bidding strategy, has a higher CTR(%) of %10.24 than Control campaign (%5.08), which is Max-bidding, which in turn found that the result is statistically significant.

• Control campaign has a higher Conversion Rate(%) of %11.4 than Test campaign (%9.2). However, there is no sufficient evidence that the result is statistically significant.

• Both campaigns have almost the same amount of purchases, Although Test campaign has a higher amount of money spent than Control campaign.

• Control campaign, which is Max-bidding, has higher impressions and reach values with regard to the aim of Max-bidding strategy.

• Overall, even if Test campaign has a higher CTR(%), it has a lower Conversion Rate (%) in terms of the mean value of performance metrics. It can be explained by marketing strategies and statistical requirements:

-The ad in campaign is targeting the wrong consumers.

-Working with a small size of data and having outliers affects the distribution, variability, and skewness with the selection of statistical methods resulting in a non-parametric test.

• It is recommended to conduct A/B test with a larger amount of data focused on targeted people for a longer period of time to observe trends, patterns, and variations. This will allow us to use of the right statistical methods to determine which campaign is more effective in a statistical manner.

Thank you for your precious time.I hope you enjoy my data world as much as ı do.
Keep grinding !