In this post, I’m going to talk about testing. In particular, an example of low-tech testing that I really did for a former employer and some of the non-intuitive things we learned without investing in a ton of technology.
Right now, you might have some part of your business that you suspect could work more effectively. How do you know if you should make a change or not? Or maybe you know something must change, but you don’t know what the new thing should look like. How do you choose?
The short answer is: data. You want to test different options in the real world in a way that will allow you to see what option works best before committing to one.
High Tech and Low Tech Testing
At a basic level, all testing works basically the same. You have at least two options and you measure how the options do against one another. Low tech testing of marketing has been going on for at least a century (depending on what kind of analysis you’re looking at). In the 1920s, for example, a number of studies were published about what coupons printed in newspapers and magazines had the highest response rates.
Over time, of course, technology has increased our ability to fine tune what we learn. In the 1940s, magazine publishers were able to more easily do “tip-ins” (ads or features or whatever that are added to a magazine, usually after it’s been bound). This opened the door for tipping in different versions of an ad to see which sold better.
In the era of the Internet, back in the early days of Amazon.com, you used to be able to hit the refresh button on your browser and see different front-page layouts. You could actually reveal the different test versions they were doing at any given time. Now, however, that testing has gotten even more subtle. But it’s still all the same idea. Does Version A perform better than Version B? Let’s try them both on real users and see!
Back to Basics: My Mail Order Coupon Tests
As some people know, I’m something of a tea geek. As such, I’ve worked for a number of different tea businesses (as well as having my own educational/consulting company related to tea). In one of the businesses I worked for, we noticed that we had quite a lot of first-time mail order customers, but a very low percentage of repeat business. It seemed like a reasonable goal to increase repeat business. The way we decided to do that was to add a coupon in every outgoing mail order package. The more coupons that got redeemed, by definition the more repeat orders we were getting, since the only way to get a coupon was to have already ordered at least once before.
We set up a baseline by sending out coupons for a month or two. Our main metric was essentially the percent of total orders in a month that used a coupon. We could have looked at how many orders were repeat orders, since at least theoretically, the coupon could have reminded people to be repeat orders without them actually using the coupon. As a measure of effectiveness of the coupon as a repeat-order-building tool, a straight percentage was going to be accurate enough and easier to calculate given our order system.
Once that baseline had been established, we started testing. Each month, I would design two coupons. Version A was the ‘control’. That is, whatever version had been performing the best so far. It was our standard. I’d also make a Version B that changed something about the coupon to see if that change increased the redemption rate. I’d make an equal number of copies of each coupon, collate them so they alternated in the stack, and gave the stack to the warehouse folks. That way as each order went out, the first one would get Version A, then Version B, then Version A, and so forth. For tracking purposes, each version got a slightly different coupon code to use when redeeming so we knew which one the customer had gotten.
What We Tested
Over the course of about two years, we tested lots of things. We tested the color of the paper it was printed on. We tested amount of the discount. We tested a discount described as a dollar amount (“$10 off your next order”) and a percentage (“10% off your next order”). We tested serif vs. sanserif typeface. We tested location of different parts, such as whether the discount information put at the top so it was the first thing made a difference compared to having it in the middle of the coupon after some kind of headline text. Font size. Line weight. Wording. All kinds of variations on all kinds of things.
After each pair of coupons expired, I’d do the math to figure out if we had a winner. Often, we didn’t. Statistically speaking, although one might have been a little ahead of the other in terms of raw numbers, it wasn’t enough of a difference. It’s what’s called the “null hypothesis.” You assume any difference is due to chance or inaccuracy of the numbers or whatever unless the difference is “statistically significant.” We had lots of null-hypothesis results. And that was important. On the one hand, we’d just spent a bunch of time testing and found, essentially, nothing. On the other, we could safely use either option knowing it wouldn’t hurt. If the boss really liked Version B much better than Version A? Great. Doesn’t matter. Go for it. We’ve essentially got proof that it’s not worth worrying about any more.
Some of these “no result” tests were still instructive, however. A 10% off coupon pulled just as many responses as one for 40% off (which, if I recall correctly, was the highest amount we tested). Why use a larger discount if it didn’t get you any more purchases? We found no significant difference between percent-off and dollars-off. But in that case it was a little trickier. The percent-off coupons resulted in larger purchases. They didn’t result in more frequent purchases which is really what we were looking for, but the average order size was much larger, so when I noticed that, we stuck to percentages.
And there were also some surprising results that did show a difference. An expiration date 2 months in the future didn’t do nearly as well as one with an expiration date 3 months in the future.
One big result was a “positive message” vs. “negative message” test. One version was something like “Thank you for your order! We appreciate your business and would like to offer you this discount on your next order to show how much we love our customers!” Something like that. Gratitude, customer-focused, that kind of good stuff. Version B was totally playing off fear and loss. It went along the lines of “You don’t want to find yourself without tea, do you? You’d better use this coupon to reorder early or you might run out!” Being negative soundly defeated the nicer version.
But the biggest, most decisive victory between versions was perhaps the least intuitive. All of our coupons for all of the tests were printed on quarter sheets (4.25″ x 5.5″). One round, I used no border. On the other, I used a dotted-line border around the outside edge. The one with the border showed a huge increase in redemption rate. Like not just 5% or 10% more, but more on the range of multiple times as many. Following that, I did lots of testing of the border, but I’d apparently hit the right one the first time. A double solid line or a fancy scroll border or any other version didn’t make a difference. But a dotted line around the outside was our golden ticket. Why? Maybe because it reminded people of a coupon you had to cut out of a magazine? I don’t know. They just needed to enter the coupon code on our website, or tell it to the person on the phone, so no cutting out needed to happen. But for whatever reason, we found success in an otherwise completely irrelevant dotted line.
What This Means To You
Think back to that thing you’re not sure what to do in your business. Apply some testing. There’s lots of online technology to help you do that, or you may be able to devise something low-tech to track, like our two coupons with different coupon codes. But do start thinking of making decisions based on data. You may reveal your own “dotted line”—that non-intuitive thing that for whatever strange reason, makes your customers more likely to buy and buy again.