Show Menu
TOPICS×

A/B testing

If you have several contents for an email delivery and you want to find out which version will have the biggest impact on the targeted population, you can send the different versions to some of your recipients, then select the one with the highest success rate and send it to the rest of your recipients.
In this use case, we are going to compare two email delivery contents via a targeting workflow. The message and the text are identical in both deliveries: only the layout changes.
The targeted population is divided into three: two test groups and the remaining population. A different version of the delivery is sent to each test group. After the delivery, a 5-day wait period is configured before collecting the results of the best open rates. The content of the delivery with the highest score is then recovered by a script and sent to the population that wasn't used as a test group.
Please note that the criteria that will decide which delivery is the best may be altered to meet your needs. It can be the open rate, the click-through rate, the subscription rate, reactivity, etc.
Moreover, the test detailed in this use case concerns only two deliveries, but you can test as many versions as necessary. Simply add activities to the workflow.
To create the A/B test, apply the following steps:

Step 1: Creating a targeting workflow

You need to create your workflow in the Targeting and Workflows tab of a campaign. It is made up of a Query activity, a Split activity linked to two Email delivery activities, a Wait activity, a JavaScript code activity, and a Delivery activity.
  1. If you haven't already done so, create a campaign (for more on this, refer to this section ).
  2. Go to the Targeting and Workflows tab.
  3. Change the label of the existing workflow or click Add to create a new one (for more on this, refer to this section ).
  4. Use the mouse to drag and drop activities into the workflow diagram, including a Query ( Target tab), a Split ( Target tab), two Email deliveries ( Deliveries tab), a Wait activity ( Flow Control tab), a JavaScript code activity ( Actions tab), and a Delivery activity ( Actions tab).

Step 2: Configuring population samples

Configuring the Query activity

  • Double-click the Query activity.
  • Click the Edit query link and select the recipients you want to target.
  • Link the Query activity to the Split activity.

Configuring the Split activity

This activity lets you create several populations: the one that receives delivery A, the one that receives delivery B, and the remaining population. Using random selection lets you target just part of the population of each delivery.
  1. Creating population A:
    • Double-click the Split activity.
    • In the existing tab, change the label to population A.
    • Select the Limit the selected records option.
    • Click the Edit link, select Activate random sampling , and click Next .
    • Set the threshold to 10%, then click Finish .
  2. Creating population B:
    • Click Add to create a new tab for population B.
    • Limit the population to 10% as previously.
  3. Creating the remaining population:
    • Go to the General tab.
    • Select Generate complement .
    • Change the label to specify that this population includes neither A nor B, and click OK to close the activity.

Step 3: Creating two delivery templates

We now want to create two delivery templates. Each template will be referenced in an Email delivery activity linked to the Split activity. For more on this, refer to this section .
  1. Go to the Resources > Delivery template folder.
  2. Duplicate the Email delivery template.
  3. Create the content to be used for delivery A.
  4. Repeat this process to create a template for delivery B.

Step 4: Configuring the deliveries in the workflow

The next step is to configure the deliveries. They are destined for the three populations created during the previous stage: Step 2: Configuring population samples . The first two deliveries enable you to send different contents to population A and B. The third delivery is destined for the population that received neither A nor B. Its content will be calculated by a script and will be identical to either A or B, depending on which one scored the highest open rate. We need to configure a wait period for the third delivery, to find out the outcome of deliveries A and B. This is why the third delivery includes a Wait activity.
  1. Go to the Split activity and link the transition destined for population A to one of the email deliveries already in the workflow.
  2. Double-click the delivery to open it.
  3. Using the drop-down list, select the template for delivery A.
  4. Click Continue to view the delivery, then save it.
  5. Link the transition of the Split activity destined for population B to the second email delivery.
  6. Open the delivery and select the template in delivery B, then save the delivery.
  7. Link the transition destined for the remaining population to the Wait activity.
  8. Open the Wait activity and configure a 5-day waiting period.
  9. Link the Wait activity to the JavaScript code activity.

Step 5: Creating the script

The choice of the delivery content destined for the remaining population is calculated by a script. This script recovers the information concerning the delivery with the highest rate of opens and copies the content into the final delivery.

Example of a script

The following script can be used as is in the targeting workflow. For more on this, refer to Implementation .
 // query the database to find the winner (best open rate)
   var winner = xtk.queryDef.create(
     <queryDef schema="nms:delivery" operation="get">
       <select>
         <node expr="@id"/>
         <node expr="@label"/>
         <node expr="[@operation-id]"/>
         <node expr="[@workflow-id]"/>
       </select>
       <where>
         <condition expr={"@FCP=0 and [@workflow-id]= " + instance.id}/>
       </where>
       <orderBy>
         <node expr="[indicators/@estimatedRecipientOpenRatio]" sortDesc="true"/>
       </orderBy>
     </queryDef>).ExecuteQuery()
   
   // create a new delivery object and initialize it by doing a copy of
   // the winner delivery
   var delivery = nms.delivery.create()
   delivery.Duplicate("nms:delivery|" + winner.@id)

   // append 'final' to the delivery label
   delivery.label = winner.@label + " final"

   // link the delivery to the operation to make sure it will be displayed in
   // the campaign dashboard. This attribute needs to be set manually here since 
   // the Duplicate() method has reset it to its default value => 0
   delivery.operation_id = winner.@["operation-id"]
   delivery.workflow_id = winner.@["workflow-id"]

   // adjust some delivery parameters to make it compatible with the 
   // "Prepare and start" option selected in the Delivery tab of this activity
   delivery.scheduling.validationMode = "manual"
   delivery.scheduling.delayed = 0
 
   // save the delivery in database
   delivery.save()
 
   // store the new delivery Id in event variables
   vars.deliveryId = delivery.id

For a detailed explanation of the script, refer to Details of the script .

Implementation

  1. Open your JavaScript code activity.
  2. Copy the script offered in Example of a script into the JavaScript code window.
  3. In the Label field, enter the name of the script, i.e.
    <%= vars.deliveryId %>
    
    
  4. Close the JavaScript code activity.
  5. Save your workflow.

Details of the script

This section details the various parts of the script and their operating mode.
  • The first part of the script is a query. The queryDef command lets you recover from the NmsDelivery table the deliveries created by executing the targeting workflow and to sort them based on their estimated rate of opens, then the information from the delivery with the highest rate of opens is recovered.
    // query the database to find the winner (best open rate)
       var winner = xtk.queryDef.create(
         <queryDef schema="nms:delivery" operation="get">
           <select>
             <node expr="@id"/>
             <node expr="@label"/>
             <node expr="[@operation-id]"/>
           </select>
           <where>
             <condition expr={"@FCP=0 and [@workflow-id]= " + instance.id}/>
           </where>
           <orderBy>
             <node expr="[indicators/@estimatedRecipientOpenRatio]" sortDesc="true"/>
           </orderBy>
         </queryDef>).ExecuteQuery()
    
    
  • The delivery with the highest rate of opens is duplicated.
     // create a new delivery object and initialize it by doing a copy of
     // the winner delivery
    var delivery = nms.delivery.create()
    delivery.Duplicate("nms:delivery|" + winner.@id)
    
    
  • The label of the duplicated delivery is modified, and the word final is added to it.
    // append 'final' to the delivery label
    delivery.label = winner.@label + " final"
    
    
  • The delivery is copied into the campaign dashboard.
    // link the delivery to the operation to make sure it will be displayed in
    // the campaign dashboard. This attribute needs to be set manually here since 
    // the Duplicate() method has reset it to its default value => 0
    delivery.operation_id = winner.@["operation-id"]
    delivery.workflow_id = winner.@["workflow-id"]
    
    
    // adjust some delivery parameters to make it compatible with the 
    // "Prepare and start" option selected in the Delivery tab of this activity
    delivery.scheduling.validationMode = "manual"
    delivery.scheduling.delayed = 0
    
    
  • The delivery is saved in the database.
    // save the delivery in database
    delivery.save()
    
    
  • The unique identifier of the duplicated delivery is stored in the workflow variable.
    // store the new delivery Id in event variables
    vars.deliveryId = delivery.id
    
    

Other selection criteria

The example above lets you select the content of a delivery based on the rate of opens of emails. You can adapt it to base yourself on other delivery-specific indicators:
  • Best click throughput: [indicators/@recipientClickRatio] ,
  • Highest reactivity rate (email open and clicks in the message): [indicators/@reactivity] ,
  • Lowest complaint rate: [indicators/@refusedRatio] (use the false value for the sortDesc attribute),
  • Highest conversion rate: [indicators/@transactionRatio] ,
  • Number of pages visited following the reception of a message: [indicators/@totalWebPage] ,
  • Lowest unsubscription rate: [indicators/@optOutRatio] ,
  • Transaction amount: [indicators/@amount] .

Step 6: Defining the final delivery

Once the script is created to select the A/B test winner, you can define the parameters of the final delivery.
  1. Connect the JavaScript code activity to the remaining Delivery activity.
  2. Open the Delivery activity.
  3. Uncheck the Generate an outbound transition option to finish the workflow with this activity.
  4. Leave the other options to their default values.
By preparing the delivery specified in the transition (defined via the Javascript Code activity), you will be then able to approve it and to start the sending, as described in the next step.

Step 7: Starting the workflow

  1. Click Start the workflow.
  2. Approve target and content for deliveries A and B via the campaign dashboard.
  3. Confirm delivery.
  4. Wait until the end of the 5-day period to find out which content was calculated after delivery opening results.
    In this case, template B was chosen.
  5. Once the content of the third delivery is determined, approve the target and the content.

Step 8: Analyzing the result

Once the test deliveries have been sent, you can check which recipient(s) they have been sent to and whether or not they were opened.
  • To find out which recipients have been targeted, open a delivery via the campaign dashboard and click the Delivery tab.
  • To find out whether the delivery has been opened, go to the Tracking tab.
  • Compare with the other delivery.
In our example, delivery B has scored the highest open rate. This means that content B will be used for the final delivery.