Show Menu
TOPICS×

Average Lift, Lift Bounds, and Confidence Interval

Reports include several data points and visualization representations that help you understand the lift bounds and confidence level associated with your activity. This helps you more accurately determine a winner.

Average Lift, Lift Bounds, and Confidence Interval

Reports include several data points and visualization representations that help you understand the lift bounds and confidence level associated with your activity. This helps you more accurately determine a winner.
This feature is available only when viewing reports in Table View. This feature is not available for activities that use Analytics as the reporting source (A4T).

Overview

The lift information in the Target reporting UI includes:
Element
Details
Lift
The large number and arrow reflect the expected value of the lift. This number is the midpoint of the range of the lift bounds. The expected lift arrow displays in grey until the confidence passes 95%. After this threshold, the arrow displays as red or green based on negative or positive lift, respectively.
Lift Bounds
This is the 95% confidence interval of the lift. It displays as a range below the average lift. See Example Calculation below for an example of how these lift bounds are calculated.
Boxplot Graph
The boxplot graph in the Target interface represents the expected value and 95% confidence interval of the success metric in question. Think of it as a graphical way to view the lift and lift bounds information.
There are a few key ways Target helps you interpret the confidence information, one of which is color. The graph displays any overlap in the confidence interval of a specific experience with the confidence interval of the control in grey, and any range of a specific experience’s confidence interval that is above or below that of the control confidence interval as green or red, respectively.
The length of the boxplot bar represents how large the confidence interval is in an easily understood way. As you collect more data in your activity, the bar shifts and changes. The confidence interval is derived from the variance and the sample size (number of visitors). The smaller the variance and the larger the sample size, the narrower your confidence interval.
Confidence
The confidence of an experience or offer represents the probability that the lift of the associated experience/offer over the control experience/offer is “real” (not caused by random chance). Typically, 95% is the recommended level of confidence for the lift to be considered significant.
The following illustration shows Lift Bounds and Confidence Level information:

How are Lift Bounds Calculated?

The lift bounds represent the 95% confidence intervals of the lift that the specific experience or offer has over the control experience or offer. Loosely speaking, it means the actual lift has about a 95% chance of being between these bounds.
The lift bounds are calculated using the following formula:
There is some additional calculation to arrive at the input to our lift bounds:
  • t-value: The critical statistic for our 95% confidence level is 1.96. You can learn more about t-values here.
  • Lift Variance: The Standard Error of Experience N’s success metric and the Standard Error of the Control Experience’s success metric are needed to determine the lift variance, which is calculated using the following formula (illustrated in the case the success metric is conversion).
  • Conversion Rate / Success Metric Standard Error: Standard error is calculated in the same way for Experience N and the Control, using the following formula (illustrated in the case the success metric is conversion). You can learn more about standard error here.
    The standard error for revenue success metric activities is based on the sample variance of the revenue.

Example Calculation

Let’s consider an example activity with two experiences and the following results:
Experience
Visitors
Conversions
Conversion Rate
Experience A (Control)
219, 328
2,466
1.12%
Experience B
218, 362
3,040
1.39%
Based on our formulas, we can calculate the inputs we need for the lift bounds.
Standard Error for Experience A (Control)
Standard Error for Experience B
Lift Variance for Experience B
Lift Bounds for Experience B
Expected Lift for Experience B:
Therefore, the lift bounds for Experience B would be:
Expect minor variances between manual calculations using the above formulas and the numbers displayed in the report. The difference can be attributed to the fact that the page views numbers used in manual calculations are rounded-off. The lift shown in the Target report is based on the exact numbers obtained from the total engagement and the engagement count. The engagement numbers can be obtained via the performance report API.

When Are Lift Bounds Not Displayed?

In certain cases, Target does not display lift bounds:
  • For any activity, when the total number of visits or visitors is fewer than 30.
  • For Auto-Allocate activities, no lift bounds are displayed until one experience has attained 60% confidence.