Create Multivariate Tests

Transcript

Hello, everybody, and thank you for joining me for this learning module on Adobe Target and creating a multivariate test. My name’s Drew Burns, and I’m a Principal Product Marketing Manager for the Adobe Target solution. So the learning objectives, after completing this training, you should be able to define and design a multivariate test and also create a multivariate test in the three-step guided visual workflow of the Adobe Target solution. So first, let’s define what is a multivariate test. A full factorial multivariate test compares all possible combinations of offers in the elements on your page to help you determine the best possible combination of content. Multivariate tests, also known as MVT, also show which element most contributes to campaign success. One advantage a multivariate test provides compared to an A/B test is the ability to show you which elements on your page have the greatest influence on conversion. This is also known as the main effect. This information is useful, for example, by helping you determine where to place content where you will receive the most attention, or what elements to focus on that will refine the experience even further. Multivariate tests also help you find compound effects between two or more elements on a page. For example, a particular ad might produce more conversions when you combine it with a certain banner or hero image. This is also known as the interaction effect.

Because of the number of combinations that can be generated, a multivariate test requires more time and traffic than an A/B test, so the page must receive enough traffic to produce statistically significant results for each experience, and this is part of the reason why we include a Traffic Estimator as part of the solution which I’ll show you in just a second, but before we move on, multivariate tests can be used together with A/B tests to optimize your page. So this is what I want you to think about is multivariate tests can either be a way to further refine an A/B test or results of an A/B test or even could be used as a discovery phase on which elements to focus on further in an A/B test. So here’s a couple of examples. You can use an A/B test to optimize your page layout first. Then follow that by an MVT test to determine the best content in each element on the page. So once I’ve understood the layout, then focus in on the combinations of elements, or we can use an MVT test to determine which element is the most important. Then, follow up with a much more detailed focused A/B test on that element. So just something to think about how multivariate tests can work alongside A/B tests. Now, with all that, with the Traffic Estimator and the traffic estimation that you have to think about, planning an MVT test is very important, and so, here’s a few things to remember. Use an MVT test when you have at least three elements to test. If you have fewer, run a series of A/B tests. Select the page elements you believe will have the strongest impact on the results. Obviously, don’t include the footer as one of the elements 'cause it probably won’t have as much impact as elements higher up on the page. Don’t include too many elements or locations in a test. The larger the number, the longer the test duration. Plan the test design in advance. Obviously, you don’t wanna be making changes on the fly, especially as an activity is running. It is recommended that elements be independent of each other. So think about this. For example, do not test your layout and content in the same test. Important to remember, and then, plan additional time, obviously, for QA because of the increased number of experiences. That’s very important as well, so with that, let’s move on to how do I create a multivariate test? So here I am. I’ve moved into my Activity Lists, which is basically the home page within the Target solution. I can come up here and create an activity, and one of the selections is a Multivariate Test. Now the great thing is if you’ve seen any of the other modules that I’ve done on these activities, once you know how to do one, you really know how to do all the rest of them because it all follows the same standard three-step guided visual workflow, and those three steps are obviously Create, which we’re doing right now, then Target, and then Goals & Settings. So in the Creation phase, I can either pick a form-based workflow or go into the Visual Experience Composer if I mull the site, and all the locations on the page are highlighted, allowing me to do, create variations and create edits on these elements. So I’m gonna go ahead and select the image here, and it’ll allow me to change that image, and I can select a couple of different options here. Maybe if you, all right, and we can even edit the links there, but if we add it, now we’ve got this location here, and once I come over, I know there’s four different variations here that I’m including, so the default and three others. So if I wanna test, let’s say, the text on the call to action, which is fairly common to go along with this, change this to Click here. I can continue to make variations as many as I’d like again, and then, I’ve got two different options here and four different options here. So in the sake of time, I could obviously add more and more. I go into the Traffic Estimator here. So it’s gonna ask me what my typical conversion rate is, and it’s also gonna ask me the number of activity impressions per day, and then, when I press Return, it’s gonna make suggestions on either limiting the amount of elements that I’m running or the amount of time that it might take. As we obviously get higher in terms of impressions, it will let you know, “Excellent traffic, good to go!” So it’s going to allow you to gauge either how long something might take in a more complex scenario but also give you the heads up that this is all set to go and will take a very efficient amount of time. So once I’ve created these variations, again, I’m able to scroll over and see how many variations there are, but really, we wanna be able to preview this. So what I can do is go through all the different variations here and see the different combinations I’ve selected, and you know what? If one doesn’t quite work for whatever reason, it’s a combination we don’t want to include, go ahead and exclude that. So it’s that easy to visually QA these and include or exclude the variations that you want. I can also, too, see what it will look like as a mobile viewport. So if I wanna see how it’ll look in a tablet or a smartphone or other custom mobile viewport, I can also set that, take a look at that, and make adjustments there. So after we’re satisfied with the different variations here, we can come in. Default, this is set for All Visitors. If I wanna come in here, I can go ahead and add a different audience segment from my Audience List, a prebuilt or ones that I’ve custom built audience segments, or ones that I’ve sourced from Analytics or other parts of the Marketing Cloud since those have the same audience segment definition as they do in Target, allowing me to take action on audience segments from Analytics, but let’s just keep it at All Visitors. Again, I can limit the number percentage of people that are gonna be included in this. I’m running a Full Factorial, and here’s all of the different variations here, and then, we move on to the third step, which is Goals & Settings. So I can prioritize this activity if I have other activities running on the page that will include the same audiences to keep mutual exclusivity really important. Set the duration here. I can also add success metrics. So these are things that I wanna track for performance purposes on the page, and it can be anything from a click-through to a next page to reaching specific experience further down the funnel to ultimately converting whatever that conversion thing is, if it’s purchasing or signing up, registering or reaching out for more information. So we’ll go ahead and select that as a conversion goal. Now, what’s interesting, if I have Adobe Analytics integrated with Adobe Target, I can just select the reporting suite that I want to apply here, and then, I won’t have to fill out anything else. So that’s it. Our multivariate test is complete. Here, I’ve got the Activity Location. I’ve got Experience URLs that I can now share to people that need to QA those experiences. I’ve got priority information. I’ve got a record of what I’ve done here. I’ve got my reporting, both a Location Contribution as well as Experience Performance. I have Collision alerts which will let me know if there are, on this page, any other activities that are running that could potentially collide, and then a Change Log for my activities. So once I’m happy with this activity, I can activate it and push it live, and that’s when we’ll start to receive different reports with live data. So if we dive in, here’s the sample of the report for Experience Performance. This is, again, where I get to see all the combinations and how they perform together in the different experiences. I could name those different experiences, too, earlier on if I wanna have terms that allow me to remember what those variations are or combinations are together. One other thing I wanna call out is I can select it to show the top five or the worst five if I have a number of different combinations. Again, as you add on more variations of elements, it goes exponentially, but if we wanna look in the Location Contribution, which, again, can be really impactful in focusing in on what element, was it the call to action, the imagery, or the messaging, what’s really moving the needle here, I could see, you know, oh, Location One, that location that I selected first! That image, that’s really moving the needle! And I really need to focus on that, and I can even drill down and see what those elements are that really caused that positive effect. So really powerful reports based on seeing not just the combination of elements and what’s most impactful for different audience segments, being able to drill down with Analytics there, but also that element contribution and what elements are really moving the needle based on those audiences. Thank you so much. -