Monthly Archives August 2014

Low-Tech Copy Testing: A Real-World Example

Antique typewriter with typed words, "Testing...testing... 1... 2... 3..."

Testing my 1924 Underwood 3-Bank Portable typewriter

In this post, I’m going to talk about testing.  In particular, an example of low-tech testing that I really did for a former employer and some of the non-intuitive things we learned without investing in a ton of technology.

Right now, you might have some part of your business that you suspect could work more effectively.  How do you know if you should make a change or not?  Or maybe you know something must change, but you don’t know what the new thing should look like.  How do you choose?

The short answer is: data.   You want to test different options in the real world in a way that will allow you to see what option works best before committing to one.

High Tech and Low Tech Testing

At a basic level, all testing works basically the same.  You have at least two options and you measure how the options do against one another.  Low tech testing of marketing has been going on for at least a century (depending on what kind of analysis you’re looking at).  In the 1920s, for example, a number of studies were published about what coupons printed in newspapers and magazines had the highest response rates.

Over time, of course, technology has increased our ability to fine tune what we learn.  In the 1940s, magazine publishers were able to more easily do “tip-ins” (ads or features or whatever that are added to a magazine, usually after it’s been bound).  This opened the door for tipping in different versions of an ad to see which sold better.

In the era of the Internet, back in the early days of, you used to be able to hit the refresh button on your browser and see different front-page layouts.  You could actually reveal the different test versions they were doing at any given time.  Now, however, that testing has gotten even more subtle.  But it’s still all the same idea.  Does Version A perform better than Version B?  Let’s try them both on real users and see!

Back to Basics: My Mail Order Coupon Tests

As some people know, I’m something of a tea geek.  As such, I’ve worked for a number of different tea businesses (as well as having my own educational/consulting company related to tea).  In one of the businesses I worked for, we noticed that we had quite a lot of first-time mail order customers, but a very low percentage of repeat business.  It seemed like a reasonable goal to increase repeat business.  The way we decided to do that was to add a coupon in every outgoing mail order package.  The more coupons that got redeemed, by definition the more repeat orders we were getting, since the only way to get a coupon was to have already ordered at least once before.

We set up a baseline by sending out coupons for a month or two.  Our main metric was essentially the percent of total orders in a month that used a coupon.  We could have looked at how many orders were repeat orders, since at least theoretically, the coupon could have reminded people to be repeat orders without them actually using the coupon.  As a measure of effectiveness of the coupon as a repeat-order-building tool, a straight percentage was going to be accurate enough and easier to calculate given our order system.

Once that baseline had been established, we started testing.  Each month, I would design two coupons.  Version A was the ‘control’.  That is, whatever version had been performing the best so far.  It was our standard.  I’d also make a Version B that changed something about the coupon to see if that change increased the redemption rate.  I’d make an equal number of copies of each coupon, collate them so they alternated in the stack, and gave the stack to the warehouse folks.  That way as each order went out, the first one would get Version A, then Version B, then Version A, and so forth.  For tracking purposes, each version got a slightly different coupon code to use when redeeming so we knew which one the customer had gotten.

What We Tested

Over the course of about two years, we tested lots of things.  We tested the color of the paper it was printed on.  We tested amount of the discount.  We tested a discount described as a dollar amount (“$10 off your next order”) and a percentage (“10% off your next order”).  We tested serif vs. sanserif typeface.  We tested location of different parts, such as whether the discount information put at the top so it was the first thing made a difference compared to having it in the middle of the coupon after some kind of headline text.  Font size.  Line weight.  Wording.  All kinds of variations on all kinds of things.

The Results

After each pair of coupons expired, I’d do the math to figure out if we had a winner.  Often, we didn’t.  Statistically speaking, although one might have been a little ahead of the other in terms of raw numbers, it wasn’t enough of a difference.  It’s what’s called the “null hypothesis.”  You assume any difference is due to chance or inaccuracy of the numbers or whatever unless the difference is “statistically significant.”  We had lots of null-hypothesis results.  And that was important.  On the one hand, we’d just spent a bunch of time testing and found, essentially, nothing.  On the other, we could safely use either option knowing it wouldn’t hurt.  If the boss really liked Version B much better than Version A?  Great.  Doesn’t matter.  Go for it.  We’ve essentially got proof that it’s not worth worrying about any more.

Some of these “no result” tests were still instructive, however.  A 10% off coupon pulled just as many responses as one for 40% off (which, if I recall correctly, was the highest amount we tested).  Why use a larger discount if it didn’t get you any more purchases?  We found no significant difference between percent-off and dollars-off.  But in that case it was a little trickier.  The percent-off coupons resulted in larger purchases.  They didn’t result in more frequent purchases which is really what we were looking for, but the average order size was much larger, so when I noticed that, we stuck to percentages.

And there were also some surprising results that did show a difference.  An expiration date 2 months in the future didn’t do nearly as well as one with an expiration date 3 months in the future.

One big result was a “positive message” vs. “negative message” test.  One version was something like “Thank you for your order!  We appreciate your business and would like to offer you this discount on your next order to show how much we love our customers!”  Something like that.  Gratitude, customer-focused, that kind of good stuff.  Version B was totally playing off fear and loss.  It went along the lines of “You don’t want to find yourself without tea, do you?  You’d better use this coupon to reorder early or you might run out!”  Being negative soundly defeated the nicer version.

But the biggest, most decisive victory between versions was perhaps the least intuitive.  All of our coupons for all of the tests were printed on quarter sheets (4.25″ x 5.5″).  One round, I used no border.  On the other, I used a dotted-line border around the outside edge.  The one with the border showed a huge increase in redemption rate.  Like not just 5% or 10% more, but more on the range of multiple times as many.  Following that, I did lots of testing of the border, but I’d apparently hit the right one the first time.  A double solid line or a fancy scroll border or any other version didn’t make a difference.  But a dotted line around the outside was our golden ticket.  Why?  Maybe because it reminded people of a coupon you had to cut out of a magazine?  I don’t know.  They just needed to enter the coupon code on our website, or tell it to the person on the phone, so no cutting out needed to happen.  But for whatever reason, we found success in an otherwise completely irrelevant dotted line.

What This Means To You

Think back to that thing you’re not sure what to do in your business.  Apply some testing.  There’s lots of online technology to help you do that, or you may be able to devise something low-tech to track, like our two coupons with different coupon codes.  But do start thinking of making decisions based on data.  You may reveal your own “dotted line”—that non-intuitive thing that for whatever strange reason, makes your customers more likely to buy and buy again.

Posted by Michael J. Coffey  |  0 Comment  |  in Analysis & Testing

Watch Out For This Problem with Facebook Ads

Since Facebook went public, they’ve been trying to make a transition.  They started as a site with its core centered firmly in a free service that replicates electronically the connections we have in “real life.”  They now need to be a site centered in what the shareholders want: money.  One of the primary ways they’ve done that is with what I’ll call “Facebook ads”—but is broader than just ads.  It’s the various ways a business (or individual, but let’s stick to business) can promote itself.  This includes paying to “Boost Post” or paying to “Promote Page” in addition to using the Facebook Ads Campaign Manager or posting Offers.

However you want to do it, you’ve chosen to drop some money in Facebook’s pocket for some extra exposure to new (or existing) people.  You go through the steps to let them know the audience you’d like to reach.  You send them the cash, and they promote you to the right people.

Or, at least that’s how it’s supposed to work.  But there’s one problem with Facebook ads: they’re not that smart.

A Personal Example: The Facebook Ads Shown To Me

Here’s a screenshot of some ads I saw last week:

Facebook ads for and the Pilot Metal Falcon fountain pen

Facebook ads I saw last week

Seems like they’re targeting pretty well.  I’ve been talking about having started a business recently, so the Customer Relationship Management (CRM) site is a good bet.  I’m also a big fan of fountain pens, so the Pilot pen is a pretty good match for me.  Not only that, but Facebook has shown me those ads not just based on a guess, but based on web data collected behind the scenes.  These are exquisitely tailored to me in particular because I’d been talking about both in private messages on Facebook.  Yes, and not just a Pilot fountain pen, but the Pilot Metal Falcon, Extra Fine Nib, with a black barrel.

The Problem

What’s the problem?  I was talking about both of those things because I already have them.  I talked about because I was asking another business owner what CRM system he used and whether he’d tried Insightly or not.  I always want to be aware of alternatives so I can change if it makes sense to do so.  And I was talking about the Pilot Metal Falcon because I was telling a friend I’d just bought it and loved how it felt while writing.  (Here’s a German word for you: Schreibvergnügen — writing pleasure.)

Pilot Metal Falcon fountain pen

Gratuitous pen shot. The soft nib allows variation in line thickness based on pressure.

Still–why does it matter?  Think of it from Insightly’s or Amazon’s point of view.  They paid to have their ads seen by relevant people.  Although Facebook did a pretty good job at picking what to show me based on what it knows about me, it was too late on both guesses by a couple of weeks.  It wasn’t smart enough to understand the context in which I was talking about those products.  It couldn’t draw the conclusion, “Michael already has those things.  Although he’s talking about them, it’s not because he’s a potential buyer.”

In other words, I’m a false positive for an ad match.

If you choose to use Facebook ads, then, you need to be aware that some portion of your money will go to advertising to people who have precisely zero chance of buying because they just got—and may be highly pleased with—your product or an alternative to it.  (I bought my pen, by the way, from The Lost Quill a small, local, woman-owned business in Bainbridge, WA, just a short scenic ferry ride and walk from downtown Seattle.  Annabella made me try different pens and inks until I was sure the Metal Falcon elicited the most Schreibvergnügen.)

The Solution

I’m all about solutions, not just complaining about Facebook and showing off my lovely pen.  Here’s what you can do to avoid wasting money: measure your return on investment (ROI).  Set up your ads so that you can tell which ad or campaign it came from–for example, give it a unique name, or if you’re using Google Analytics, be sure to use UTM tagging on your link.  That way you have a chance at seeing how many people came to your site because of the ad. Depending on your tracking options and the goals for the ad, you should be able to determine how many of that set of visitors ended up converting.  (A conversion could be buying, creating an account, downloading, subscribing, watching a video, or whatever action you are looking for).  Then you can see whether the cost of getting that conversion was worth what you paid for it.

And here’s the best part: whether it was worth it or not, it doesn’t matter how many of the people who saw it were false positives.  If Insightly got enough paying members because of that ad that it was profitable to do so, it doesn’t matter if there were 10,000 false positives in addition to me.  The end result is the same: profit.  On the other hand, if Amazon finds that they didn’t sell enough pens with their campaign, it doesn’t matter if I was the only false positive out of 10,000 people.  It still wasn’t worth it.

The moral of the story, then, is twofold:  First, you should know that Facebook ads sometimes (often?) are shown to inappropriate targets.  Second, if you set up your ad campaign and web tracking correctly, the first item can be safely ignored.

If you have specific questions related to this, please feel free to contact me and I’ll do my best to answer them.

Posted by Michael J. Coffey  |  0 Comment  |  in Social Media

Thinking Critically About Social Referral Data

What should you do for the best social referral data? Like the picture says: Test, test, test!

A couple of weeks ago, Shareaholic published its social media traffic report for the second quarter of 2014.  In it they looked at social media data from 8 of the more well-known sites:  Facebook, Pinterest, Twitter, StumbleUpon, Reddit, YouTube, Google+, and LinkedIn.  Based on their social referral data, which come from over 300,000 websites, Facebook drives the most traffic to websites, hands down.  Like nearly 25% of all visits.  You might then immediately jump to the conclusion that your business needs to be on Facebook because it’s the best.

Let’s assume the numbers are correct and accurate.  It’s not something I always assume, but in this case I want to thinking about what they mean.  To do that, we have to think about the situation that might have contributed to those numbers.  How do people on those social media sites end up on the various websites that they measured?  In other words, what is the pathway that is being measured?

A Visitor’s Path

The most likely way, of course, is the social share.  Someone shares a link to one of the websites in a social media post.  You know, probably with a comment like, “OMG. This is so cool, you have to see it!” or “Great article” or whatever.  Their friends/followers see the post, click on the link, and *ding*!  They’ve been tracked as referral traffic from a social media site.

So far, so good.  The unsuspecting business owner still says, “Great.  If I’m on Facebook, more people are there, more people will see the post, and more people will click on the link and visit my site!”  Eh.  Not so fast.

The Other Half

If we apply systems thinking concepts to this, we can recognize that we’re only looking at the second half of the full system.  At this point we haven’t looked at how and what gets shared, and that’s where some issues start to be revealed.

Many sites include social sharing buttons.  You know, the little buttons with all the social media sites where you can just hit it to create a new post with the link to that article already filled in.  A person reading an article on a site might copy the web address and paste it into their own post manually.  But many people will use those buttons to share.  And here’s where we start really thinking critically: how much influence do those buttons have on the referral traffic back to the site?

I’ve been on many sites that only have a Facebook and a Twitter button.  I know that sometimes I want to share the article on Google Plus, and have in a few cases, not shared the article because there was not a G+ button.  As a result, my G+ followers didn’t see a link to the page, meaning they didn’t follow it and become a social-referral visitor.  Meanwhile, Facebook and Twitter are getting higher amounts of exposure because of the buttons, thereby increasing the social shares on those sites.

So here’s the thing:  Is Facebook really driving more traffic, or are websites just more likely to have a “Share on Facebook” button?

We could even go further with this…for example, some social share systems have a place where you can click and reveal a button for all the sites you’ve never even heard of.  But data shows the more clicks you need to do, the fewer people will follow through.  And people are more likely to click the first item than the second one, and so forth.  So even the order of the buttons might make a difference in how and where an article is shared.

The Answer: Look at Your Social Referral Data

The answer to this confusion brings us back to the image on this post:  test, test, test!  The only way to really be sure for your website is to look at your own visitor data.  Just now, I checked the last two months of data for and my other site,  One site got a strangely-round number of 60.0% of social media referral traffic from Facebook.  The other got 9.86% of its social referral traffic from Facebook.

Looking at Shareaholic’s article, you might be tempted to think you have to be on Facebook.  In fact, they say in a bold header, quite definitively, “Facebook is, by an extremely wide margin, the #1 source of social referrals” But I’ve got a site where over 90% of social referrals come from not Facebook.  It gets roughly three times as much traffic from StumbleUpon than Facebook, but from Shareaholic’s standpoint, Stumble upon only drives a measly 0.6% of traffic.

Who am I to believe?  Well, my own numbers, of course.  It doesn’t matter what the average is.  It matters what is working for you.  And if you don’t like what you’re getting, experiment with new things and see if you can improve your results.  That’s why systematic experimentation and testing is usually part of my digital strategy recommendations.  You can’t tell what results you’ll get by looking at averages, and you can’t tell what will work best unless you try different things and measure your outcomes.

Which sites work best for you?  (Or, if you’re not sure, what challenges do you face finding out?)  Leave your answer in the comments.

And don’t forget to share this article on social media by using one of the buttons!

Image source: my own notebook where a friend and I were testing different fountain pens and different inks.  Not social referral data, perhaps, but testing nonetheless.

Posted by Michael J. Coffey  |  2 Comments  |  in Analysis & Testing

What is a Business Blog For? Choose a Strategic Goal!

Kristian Berge creating content, like you should for your business blog.“Every business should have a blog!” Lots of marketing people, business consultants, SEO experts, and more have said this—maybe even to you. Heck, I might have said it, although probably not in so stark a way for reasons you’ll see in a minute. It’s a common belief, even among people who don’t have one, that a blog is necessary for business. And it is…and it isn’t. That’s why I wrote this article: to explain the common question, “what is a blog for?” After all, you don’t want to start a business blog unless you know why you’re doing it, and what it’s supposed to do for your business.

First, What Is Blogging?

A brief introduction for those who have heard the term but are a little fuzzy on what it means: a blog (short for “web log”) is like a journal, only public.  It can be like a personal journal where you write your thoughts and experiences.  It can also be like a professional journal where articles on a particular theme or subject are published.  There is software that can be installed on your own server, or you can use software that’s installed on someone else’s.  But in either case, you would use this software like a word processor to write your articles and then “publish” them—put them on the Internet for visitors to see.

From a business perspective, this gives you another online “place” for people to see your business, and for your expertise to shine or your marketing message to be shared.  To “do” blogging, you just need to write things, create imagery, or both.  When you have created that content and published it on your blog, you are blogging!

Crucial: Goals for a Business Blog

The answer to “what is a blog for” really has to do with your goals.  Don’t create a blog until you figure out what you want it to do for your business because making the right choices about how you set it up and what you publish will depend on the goal.  As Lewis Carroll put it in Alice in Wonderland:

“Alice came to a fork in the road. ‘Which road do I take?’ she asked.
‘Where do you want to go?’ responded the Cheshire Cat.
‘I don’t know,’ Alice answered.
‘Then,’ said the Cat, ‘it doesn’t matter.”

All of the following can be valid goals for a blog.  There may be more, but if any of these align with what you’re looking for, great.  You can just pick from the list.  Here’s a caution, though: Don’t Pick More Than 3.  The fewer goals you have, the more focused you can be.  The more focused you are, the fewer things you need to do to be successful.  From a strategic point of view, then, you’re better off not saying, “They all look great!  Those are all my goals!”  Rather, you should be thinking, “My blog exists to do X, so I will stop worrying about doing anything that won’t get me closer to that end.”

List Building:  Perhaps you’ve decided that a key to your success is having a large database of potential customers.  You want to use your blog to draw in new people and encourage them to subscribe to your newsletter or create a free account.  (Having a big list of potential customers is generally a good thing for all businesses, but is increasing the size of your list your primary focus?)

Establishing Credibility:  Are you new to the industry?  In a new business?  Publishing things that show off that you know what you’re talking about can improve your perceived expertise or credibility.  If you want your potential clients, or even the media, to feel comfortable contacting you as an authority in your subject, this could be a good goal.

Search Engine Optimization:  There are quite a few ways that a blog can help you show up higher on the search engine results pages, from  (because really, how often do you even see page 2?)  But worrying about SEO isn’t a very good choice if most of your clients find you through word of mouth.  That’s an entirely different type of marketing and spending your time on SEO would be a waste of effort.  And if a web search is where your likely clients find your business, it might be effort well spent.

Advertising:  If you’re selling ads on your site (or using a service like AdSense to do the selling for you), you’re interested not in subscriptions or credibility but raw traffic so that there are more people looking at your ads.  There are quite a few people whose entire business is this one goal–post blogs that attract traffic and earn money when some of them click on the associated ads.  That could be your business model, or maybe you just want to supplement your income from something else your business does.

Education:  Are there things you need people to know?  A goal could be to educate your audience—about your product, an issue, a topic, or whatever.  Perhaps you post educational videos, or how-to articles because your business runs smoother (or you reduce the cost/effort of customer service complaints) if your clients know this information before they start working with you or after they’ve made a sale.

Direct Sales:  Each post can be an attempt to sell something.  Developers have created ecommerce plugins for many blogging platforms to make this easy.  There are certainly challenges to this approach because it can come across as high-pressure sales and turn people off, but with the right offerings and the right target audience, it can work.  (I’m thinking of the start of which had only one product—a different one each day—at a great price.  It had its fans coming back every day to see what today’s product was.)

Assisting Your Sales Process:  Even if you don’t want the blog to sell directly, it can be used to move people along your sales funnel.  This could be done using many of the other goals as needed; for example, a sales person might send a potential customer a link to a blog article that addresses one of their concerns.  That’s assisting sales, but could also be education or establishing credibility, depending on the concern in question.  The focus, though, is slightly different.  You would be paying relatively more internal attention to what would be helpful for your various sales stages rather than the external “what do people on the Internet want?”

Building Relationships:  A large number of businesses are recognizing the importance of building relationships with past, current, and future customers.  A blog can act as a kind of forum for building a community.  That community and relationship building can happen through conversation in the comments on a post, or across blogs and other online media (a reader of your blog could link to your article from a post on their own blog, for example, or the post could spark social media conversation).

There we go!  I hope that gives you some ideas about what a blog can be used for in your business.  Once you have that in place, you have a guide to narrowing your options and eliminating a good deal of information overload about your online marketing.  “Does it help me do X?  No?  Then I’ll ignore it because it’s the wrong path.”  A good strategic choice.

Image source:

Posted by Michael J. Coffey  |  0 Comment  |  in Blogging
  • Stay Connected