Adapting to Changing Consumer Behaviors with A/B and Multivariate Testing

The Importance of Testing 

Consumer behaviors and preferences are everchanging and brands need to keep up with them through continuous experimentation and testing. The beauty of testing is that it can be done on different scales, from piloting a new loyalty program in specific markets to more tactical testing to determine what types of rewards would be most redeemable. Test-and-learn strategies help back hypotheses with data and allow brands to make informed decisions. Today, with the advent of automated testing and AI, the process of testing has become even more effective and reliable. Three key benefits of testing include:

The two most common ways to test are A/B testing and multivariate testing.

A/B testing compares two variations (Version A vs Version B) of the same element to determine which version generates better results. This is a relatively easy way to implement testing, but its design also means that you can test only one element at a time. If there are several elements to be tested, then this way of testing can be very time consuming. In such cases, multivariate testing is a more efficient method of testing. Multivariate testing compares multiple variations of several elements to determine which combination produces the best results. Brands can leverage both types of tests depending upon the situation and the end goal of performing the testing.

Though testing sounds simple enough, many brands make mistakes in how they determine what to test and how they finally implement it. Below, I have outlined the best practices for testing to avoid these pitfalls:

1. Do your research: A detailed assessment of what is working and what is not in your loyalty program is the first step in the testing process. Review your loyalty program from all angles: the customer experience, competitive advantages, financial performance, and business performance. Perform a detailed qualitative and quantitative analysis to identify which areas need improvement. Is the new member acquisition rate low? What is the bounce rate or abandonment rate at each stage of the loyalty cycle? Do your research to come up with these questions and design a test to get the answers.

2. Decide what you want to test: The research above will help you identify multiple areas of enhancement, however, deciding what to tackle first can be challenging. You need a structured system to prioritize the list. This process of prioritization will rank each item in terms of its level of impact and the level of effort required for testing. It is wise to begin by testing an area with a low level of effort and a high level of impact. Use the quadrant below as a guide for prioritization.

3. Define your hypothesis: A hypothesis is a prediction about the impact of the test on the success metrics of the loyalty program. It is critical to define the hypothesis concretely to run the test effectively. For example, based on the quantitative data collected and analyzed, we may observe that potential members abandon the enrollment process after they click through the CTA. We would hypothesize that adding the benefits of joining the program in a clear and concise way can improve the conversion rates by 20%

4. Determine the sample size to achieve statistical significance: One of the biggest mistakes that brands make during testing is choosing an incorrect sample size which leads to insignificant results. Use a statistical significance calculator to determine the correct sample size. As a best practice, ensure the sample size can achieve at least 95% significance level. For this to happen, you might have to run the test for a specific amount of time, which can be provided by the statistical significance calculator.

5. Create test variations based on the defined hypothesis: Depending upon what element or elements you are testing, create and design the test variations. For an A/B split test, there will usually be two variations: one would be the control and the other would be the challenger. Be sure to test only one element at a time while running the A/B split test. For multivariate tests, the test variations will include different combinations of the elements being tested. As a best practice, limit the number of variations for a clean read of the metrics and declare a winner.

6. Derive insights and learnings from the tests: The results of the test will help you understand which variation was a winner with statistical significance. Use the learnings to gain valuable insights about your loyalty program and audience behavior. This can be used further to develop new tests and experiments for continuous learning and enhancements.

What can you test for in loyalty programs? 

For loyalty programs specifically, the three key stages are acquisition, issuance, and redemption. There is opportunity to test different elements at each of these stages to ensure success. A few examples include:

• Acquisition tests: Acquiring members to become part of your loyalty program is essential for a successful loyalty program. There are many elements that need to align perfectly to motivate a consumer to become your member. Brands should test different variations of these elements to determine what resonates the most with newly acquired members. These include the benefits of joining the program, the ease of signing up and amount of data needed to sign up, the placement of this value proposition on the website, and the design of the sign-up page.
• Issuance tests: Once they join the loyalty program, brands need to encourage these new members to perform the various transactional as well as non-transactional activities that will help them earn the loyalty currency. Brands can test the type of activity that is likely to be completed the most, issuing points vs. issuing cash + points and issuing points vs. issuing cash benefits.
• Redemption tests: In the redemption stage, brands can test a variety of different types of redemption. For example, they can test to see whether discounts, experiential rewards, or a combination of both is preferable to members, as well as if accumulation of points or instant rewards are preferred.

Designing and executing a well-thought-out test plan is critical for the success of a loyalty program, but you do not have to do all the heavy lifting. Our loyalty and marketing strategists are here to help you at every step of the way. Contact us today to design and execute a test strategy.