A/B Testing
What is A/B Testing?
A/B Testing is a strategy used by social teams to see what kinds of posts resonate best with their audience. You can test out how changing a variable like your share message, title, description, image or video engages your audience most effectively.
Our clients get on average 25% more clicks when A/B Testing. Each Facebook page has its own distinct audience, which makes A/B Testing so valuable. Strategies that work well for one Facebook page might not necessarily work for another page.
- Rapid CTR calculation - Echobox determines the click-through rates for each variation, then automatically shares the winning variation with your full audience.
- Rigorous scientific standards - Our A/B Testing allows for the most scientifically rigorous hypothesis testing. View statistically significant results in our analytics to drive your long-term strategy.
How does A/B Testing work?
Echobox will share both variations of your post with a small subset of your audience for a short amount of time (up to 60 minutes). We use Facebook’s targeting to identify comparable target segments to which we share both variations independently.
These posts will not show on your page: you create them in your Compose Box and then Echobox posts them only for a subset of your audience.
Echobox then deletes the test variations from your Feed altogether and calculates the performance for each variation and determines a winner. The winning variation is then posted as a normal post for all of your audience.
How can I use A/B Testing?
A/B Testing is an opportunity for you to display creativity in presenting content to users in a way that encourages them to click. Our organic A/B Testing solution helps you understand how your Facebook audience perceives your strategy.
Some specific ideas include:
- Images and Videos - Are black and white images more effective than color? Are shorter or longer videos better?
- Tone of Voice - Should I write rather playful or more factual headlines?
- Hashtags/Mentions - Should I use #hashtags? And what about @mentions?
- Emojis - How does my audience react to emojis? 😀
Answer these and many more questions about your Facebook audience.
Strategies for A/B Testing
A general guideline for an effective A/B Testing strategy could look like the following:
- Come up with a hypothesis
- Test this hypothesis with a large number of posts ensuring that versions A and B are consistent across tests (10-30 posts should be sufficient)
- Add labels to the test posts in your Compose Box so they can be easily identified
- Export the data in Analytics and analyze the differences
To be able to perform A/B Testing, your account must be domain verified. Please refer to the domain verification guide here for assistance.
How do we judge which is the best performing post?
- Link: the click-through rate (CTR) on the link
- Photo: engagement with a photo (this is defined as a click anywhere on the post, including likes, comments, and reactions)
- Status: clicks on the post (this is defined as a click anywhere on the post)
- Video: view rate of the video, which is determined by the number of times your videos are played for at least 3 seconds, or for nearly their total length if the video is shorter than 3 seconds
Statistical Significance
To document the results of your A/B Testing, we have created a strategy template to determine whether the combined results of your tests are statistically significant. If they are statistically significant, then the best performing version will be shared.
FAQ
0% CTR in Analytics, what does this mean?
When the AB test results in the analytics section both show 0% CTR that means that the test failed to complete and Version A has been shared.
AB tests can also fail if not enough followers have seen the posts to create a statistically significant result. This is not uncommon with smaller Facebook pages (below 200,000 followers).
As a rule of thumb, it is best to place the version that you are leaning toward (from an editorial perspective) as Version A in case the test cannot be completed.
Why can’t I use Specific Time or Share Now with my AB Test posts?
The time that Echobox shares your post is based on a variety of factors: when the audience segments have seen the initial posts, when the CTR analysis is complete, and when the winning post is then shared with all audiences. Since this process takes time to complete, it is not possible to specify the share time for the post manually.
What do my A/B Testing results look like in an Analytics Report?
When you export the CSV of your analytics from your Echobox, you can make a copy of this spreadsheet template we have created for you. Copy the CTR data from your CSV and replace the values in the template with your test values. On the right of the template, in the green box, you will see your confidence interval conveniently calculated for you to analyze the effectiveness of your test.
What happens when I select both AB Test and Image Overlay in the Compose Box?
When image overlay is selected, it is the only variable that can be AB tested. To get results from your test, it is important to only test one variable at a time. Changes to the post can be applied before Image Overlay is applied. These changes will be duplicated across both versions with the only variation being the presence of the overlay.
When both Image Overlay and A/B Testing are ticked:
Version A: original image + the overlay
Version B: original image
