e-Appeals: Create New e-Appeal Part III - A/B Testing

e-Appeals: Create New e-Appeal Part III - A/B Testing

1. On Tab #3: A/B Scenarios of creating a new e-Appeal, you will select the content that will be sent to your recipients.

   

2. Be sure you have completed Tab #1: Setup and Tab #2: Segments (recipients) prior to navigating to Tab #3.
3. We also recommend that you configure your appeal template(s) prior to navigating to this tab as well, though you can configure/set up the appeal template from within the draft e-Appeal as well.
4. Tab #3: A/B Scenarios gives you the opportunity to perform "A/B" testing on your recipients; that is, to test out how your recipients respond to various different variables such as different email subject lines, different email content, different sender names, and different send times.
5. For example, perhaps you have two versions of your appeal letter. The one you've always used that is "adequate"/good, and a new one you drafted this year. You don't want to rock the boat and risk losing donations by sending the new, untested letter, but, on the other hand, maybe the new letter might be more effective. A/B testing would allow you to send both letters, with half of your recipients receiving one letter and the other half receiving the other letter. Then you can see/assess response rates. Which letter raised more money? Your recipients will be randomly assigned to one group or the other/to receive one letter or the other so it will be completely randomized sampling/the recipients will be randomized so that the test/experiment will be unbiased.
6. Or, perhaps, you aren't sure what the best time of day is to send your email; maybe you think it might be better to send it early in the day so that your donors will see it before their email boxes fill up (and your email gets buried) but also that you want to send it late enough that people are actually in their office/at their computers. With A/B testing, you can send your appeal to half your recipients at one time and the other half at a different time and then review the metrics afterwards to compare open and read rates between the two batches.
7. A third example of how to use A/B testing would be if you aren't sure if your recipients will respond to a "click bait-y" type subject line expressing urgency to open your email (e.g. "Immediate Attention Required!"). It might be, instead, that your recipients mistake it for spam and ignore it. With A/B testing, you could send half your recipients an appeal with a more sedate subject line and the other half of your recipients your more "exciting" subject line and then compare the metrics for open rates to see which was more effective.
8. Lastly, you can use A/B testing to send different content to different segments/recipients. In Tab #2, we mentioned creating a segment because you want to send those who live locally a slightly different email appeal in which you invite them to attend a reception/event (or perhaps you want to group your recipients by relationship manager and send a letter signed by relationship manager X to those they have a relationship with). With A/B testing you can send all of the recipients in one segment one email template/letter and the recipients in another segment a different letter. In addition to randomly assigning recipients to test variable (content, subject line, sender name, and send time) groups, you can also manually assign the percentage of recipients in each segment to test variable groups.
9. To get started configuring your A/B content, select which variable you would like to test: email subject line, sender name, template content, OR send time. You can only test one variable per appeal. You must select a variable. If you do not want to do A/B testing/if you want all recipients to receive the same content from the same sender with the same subject line at the same time, pick any variable here/the variable does not matter.

   

10. By default, Template is already selected as the variable to test, but you can switch to any of the other options by clicking the + button on the item you want.

    

11. After selecting a variable, select how many options you want to test by clicking the + (and/or minus) sign to increase (decrease) the number. For example, if you have two different subject lines you want to try out on your recipients (with half of your recipients getting one subject line and the other half getting the other), you would select "2" under Subject Line, as pictured below. If you wanted to try out three different subject lines, you would select "3" as the quantity. If you do not want to do A/B testing/want all recipients to get the same content at the same time from the same sender with the same subject line, select "1" for the quantity.

    

    

    

12. After selecting your variable and the quantity to test, complete the Common Info section. The fields in this section based on your variable selection.
    • If you selected Subject Line as the variable, complete Sender Name, Template/Letter content, and the send date and time of your appeal.

      

    • If you selected Sender Name as the variable, complete Subject Line, Template/Letter content, and the send date and time of your appeal.

      

    • If you selected Template Content as the variable, complete Subject Line, Sender Name, and the send date and time.

      

    • If you selected Send Time as the variable, complete Subject Line, Sender Name, and Template Content of the appeal. Please Note: Send Time is the date/time the e-Appeal will BEGIN PROCESSING to be sent. Depending on the number of recipients in the e-Appeal (and the amount of jobs in the bulk processing gateway ahead of the e-Appeal), it may take anywhere from thirty minutes to HOURS for the entire e-Appeal to process and send. If you need this e-Appeal to arrive at a certain time/within a certain time frame, be sure to factor in processing and sending time and add a little padding to the send time. Please Additionally Note: While send date cannot be in the past, the system will allow you to schedule a time in the past, in which case, the e-Appeal will begin processing immediately as soon as you submit it for processing/click send.
      

      

13. Please Note: On any of the above steps, if you select a template and then edit it, the thumbnail may take a few minutes to update.
14. Next select how you would like your recipients split into the test groups. Your recipients will be split into your test groups based on percentages. That is, if you are testing two subject lines, you will have two test groups. If you have one segment of recipients, a certain percentage of the recipients in that segment will receive subject line #1 and the remaining recipients will receive subject line #2. If you have two segments/recipient lists, then a certain % of segment #1 will receive subject line #1 and the rest of segment #1 will receive subject line #2. A certain percentage of segment #2 will also receive subject line #1 and the remaining recipients in segment #2 will receive subject line #2.  You can set whether you just want the computer to split all recipients in each segment equally among the variables or if you want to control the percentages. For instance, if you have two segments and you want 100% of the recipients in segment #1 (non local donors) to get your template/letter without the invitation to a local event and 100% of the recipients in segment #2 to get the letter WITH the event invitation text, you would select "I'll do it manually" here and then configure the percentages below. Otherwise, if you want truly random testing, simply select Split Equally.

    

15. Next configure your test variable content. Which variable you configure here will depend on which item you selected in Step #9. In addition, how many columns display here depends on the quantity selected in Step #11. If you selected a quantity of "1" you will only have one column. If you selected quantity of "2," you will have two columns. And so on.

    

    

    • If you selected Subject Line as your variable, enter your various subject lines into the Subject Line field(s) in the Test Variable section (for instance, if you have selected quantity 2 under Subject Line in steps 9-11, then enter the two subject lines you want to test here. In one box/the first field on the left, put your first subject line. In the second box/the field on the right, put your other/second subject line).

      

      

    • If you selected Sender Name as your variable, enter your various sender names into the Sender Name field(s) in the Test Variable Section.

      

      

    • If you selected Template Content as your variable, enter your various templates into the Template Content field(s) in the Test Variable Section.

      

      

    • If you selected Send Time as your variable, enter your various send dates/times into the Send Time field in the Test Variable Section. In the example below, we're testing if our recipients are more likely to open our email on a Wednesday morning or a Friday morning (both dates have the same send time).

      

      

16. Lastly, configure your Split Percentages. This is the percentage of recipients in each segment who will go into each test group. If you only have one test item (selected quantity 1/are not doing A/B testing) then 100% of all segments will get the same subject line/template content/sender name/send time. If, however, you are testing 2 more more variations of your variable, then each segment will be split across the variations. In the example below, we are testing two different subject lines for our email/appeal. We've split Segment #2, which has 31 recipients in it, 50-50 between the two subject lines so that half of my recipients in that segment will receive the the email with the subject line "Please Support MCA" and half will receiving the email with the subject line "Immediate Action Needed."
    

    

17. If you selected "Split Equally" in Step #13, then your recipients will have been randomly split into test groups of equal size. You can use the up and down arrows to change their percentages if you wish by EITHER changing the overall percentage of recipients assigned to each group OR changing the % of recipients in particular segment(s) that receive a test variable.

    

    OR

    

18. Your row/horizontal sub-totals must equal the total number of recipients in each segment when you are done (that is, all recipients must be assigned to one of the test groups).

    

19. After configuring your test variables and your split percentages, select Save in the lower right to save your changes and exit out of the e-Appeal (you can return later) or Save and Proceed to move to the next tab.