In this article:

1. Setting up an A/B test
2. Tracking your A/B test
3. Viewing completed A/B tests

A/B tests allow you to compare two slightly different versions of content to see which performs better. You can easily run A/B tests on your contribution forms using ActBlue. Our A/B Test Tool randomly splits everyone who clicks on the link to your contribution form into two groups. One group receives the normal form (the control), and one receives the altered form (the variable). The variables you can test include: 

• Form title
• Donation ask
• Preset amounts
• Branding
• Hide thermometer
• Hide embedded video
• Recurring promotions, like a recurring pitch or recurring incentive
• Pop-up recurring title
• Pop-up recurring ask
• Smart Boost upsell

You have multiple optimization options for running a test without missing out on one-time, recurring, or Smart Boost contributions.

Once the test has run its course, you can compare the two versions and see which one received more dollars and donations.

Setting Up an A/B Test

On the Contribution Forms page, click “Edit” to the right of your contribution form. 

Select “A/B testing” in the form editor.

 

Click the link in the “A/B Testing” tab.

You’ll land on the A/B Test page, where you can set up your experiment.

Give your test a descriptive name. The name may show externally, so choose something that is not sensitive or private.

Next, add a short hypothesis describing what the outcome of your A/B test will be. For example: “We expect the form with the custom title written by the content team to have more form conversions.”

Underneath these two fields, the “Automatic traffic allocation” option will automatically be selected (pictured above). This feature, also known as a multi-armed bandit algorithm, directs more traffic to the version of your form that is performing best. Depending on what you optimize for, you won’t lose out on potential contributions and can still run a sound test. We highly recommend keeping this feature on to run tests and raise funds simultaneously. You can learn more about the multi-armed bandit on our blog. 

Next, you can select the outcome you want the algorithm to optimize for while running your test. 

If you select “ Form conversions,” Automatic traffic allocation will direct more traffic to the version of your form that's collecting the most contributions of any amount. If you select “Recurring conversions,” the algorithm will direct more traffic to the version of your form that collects more recurring contributions, whether donors make them on the form during the donation flow or after, through the recurring pop-up upsell. If you select “Recurring upsell,” the algorithm will direct more traffic to the version of your form on which more donors are choosing to start a recurring donation after completing an initial one-time contribution. If you select “Smart Boost,” the algorithm will direct more traffic to the version of your form on which more donors are choosing to increase their initial contribution in the post-donation upsell. You cannot change your optimization after the test begins.

If you uncheck “Automatic traffic allocation," each form variant will receive the same traffic volume, even if one performs poorly. Please note that even if you have “Form conversions,” “Recurring upsell,” “Recurring conversions,” or “Smart Boost” selected,  each version of your form will receive the same volume of traffic if you uncheck the “Automatic traffic allocation” box. 

Next, select the feature that you would like to test:

This feature is your variable. Each form version in the test will be the same, except for the variable you choose to test. You can test multiple features simultaneously, but to run the most accurate test, just pick one. That way, you will know the variation caused a difference in visits, contributions, contribution amounts, recurring contributions, or Smart Boost contributions.

Different optimizations allow for different element testing:

• Form conversions: Test any element of your form except recurring promotions, and you cannot test any part of the recurring pop-up. 
• Recurring conversions: Test any element of your form or recurring pop-up. 
• Recurring upsell: Test the recurring pop-up title and ask. 
• Recurring promotion: Test a recurring pitch or recurring incentive
• Smart Boost: Test the Smart Boost pop-up title and ask. 

Once you’ve selected a feature to test, update the variation of that feature as needed. For example, to test the form title, your current title will populate, and you’ll need to enter an additional title to test (pictured below). 

If you choose “Recurring promotion,” you can test different variations of the recurring pitch and recurring incentive tools. The control is your form’s current settings, whether a recurring pitch or incentive (configure both in the recurring settings tab of the form editor). 

For the variation, you can test a recurring pitch or recurring incentive. That means you can test the same type of recurring promotion with different copy or a different type of recurring promotion.

If you want to add another variation, click “Add another variation” and repeat the process.

Once you’ve finalized your test variations, click the “Create Test” button. Congrats — your test is live. You’ll land on the Active A/B Test page, where the results and analysis of your A/B test will live.

You can now view both variations of your form in the form editor. Go to the “A/B testing” tab and select which variation you want to see from the “Preview A/B test variation” dropdown menu.

Tracking Your A/B Test

Once you create the test, you can track each variation’s progress by returning to the “A/B testing” tab of the form editor and following the link.

 

You will see the name of the A/B test, which admin created it, what it is optimizing for, and the hypothesis.

You can choose to receive a weekly email update about your A/B test.

Next is an analysis of the control and each form variation. First, you’ll see the traffic allocation: the percentage of traffic visiting that form version based on the multi-armed bandit algorithm. If automatic traffic allocation is off, each version of your form will have equal traffic allocation. You will also see the total money raised on that form variation.

 

No matter what you optimize your test for, you will always see the form conversion rate (people who donated divided by people who only visited the form), average contribution, and dollars per visit for each version of your form.

If optimizing for recurring conversions, you will see:

• Recurring conversion rate: percent of donors who visit the form and make a recurring donation during the donation flow or via a pop-up upsell.
• Average recurring amount

 

If optimizing for recurring upsell, you will see:

• Upsell conversion rate: percent of donors who see the recurring pop-up after giving a one-time contribution and decide to start a recurring donation
• Recurring conversion rate
• Average recurring amount

If optimizing for Smart Boost, you will see:

• Smart Boost conversion rate: percent of donors who see the Smart Boost pop-up after contributing and decide to increase their donation
• Average Smart Boost contribution: the average amount that donors decide to increase their contribution
• Dollars per visit
• Form conversion rate
• Average contribution

The Active A/B Test page does not include stats on Donors who contribute through an ActBlue Express Lane link.
If you did not use automatic traffic allocation, click “Make winner” on the top-performing variation to stop allocating traffic to the less successful form. If you use automatic traffic allocation, the A/B test will do this on its own over time, but if you see a clear winner, you can click “Make winner” to speed up the process.

You can find data on your A/B test in the spreadsheet form contributions, which you can find on your form’s Statistics page. Navigate there from the Active A/B Test page while previewing your form or by clicking either the number of contributions or the donation total for your form on the Contribution Forms page of your Dashboard.

Download the report by clicking "Export all contributions" to see contribution and A/B test data.

Viewing Completed A/B Tests

Once you complete your A/B test, either by letting the multi-armed bandit algorithm pick a winner or picking a winner yourself, you can view the results on the A/B test page of your form. You can always navigate to the A/B test page through the link in the form editor or the Form Tools menu.

If you have any questions, please contact our support team using the email address you use to access your ActBlue account.