“A man with a watch knows what time it is. A man with two watches is never sure.”

—Segal’s Law

Can You Show Me How To Do Data Analysis?

Recently, a person I worked with at a previous job contacted me to ask if she could take me to lunch so I could explain Google Analytics to her.  I was happy to help, but also wanted to help set expectations.  I pointed out that there were thousands of different reports that were possible in GA, and that was even before doing segmenting of visitors, comparing different time periods, and other more advanced ways of looking at the data.

The same week, a client asked me to show him how to interpret a report on Google Analytics.  That one report turned into almost an hour of conversation as we discussed some of the nuances that might be going on in the background behind the chart we were looking at.

Those conversations got me thinking about some of the things that seem obvious to me about looking at website data that don’t necessarily jump out at entrepreneurs who are just getting started marketing online.   One of those things is that the data is rarely clear cut.  Allow me to explain…

A Simple Example of Problematic Data

I track “reach” for my own social media marketing (among other things).  That is, how many times has someone seen something from my brand on social media.  I use Google+, Twitter, and Facebook, and all three have a way of showing what the reach is for your business account.  Easy, right?  Just look up the data for each site, compare them, and then I know how much reach I had on each site.

Au contraire, mon ami!

I bet at this point you expect me to talk about how each site measures reach a little differently and so they can’t be compared, right?  But no!  It’s even more complicated than that.

When I go to Google My Business (the dashboard behind the G+ business page) for my tea-education brand Tea Geek, this is what I see:

Easy enough.  My all-time views are 313,846.

But hold on.  Before the site transitioned to the “new” Google+ format, you could display your views on your profile itself.  I switched back to the “classic” view and saw this displayed on the page:

My all-time views are 474,236.  That’s basically 50% higher.

Let that sink in for a moment.  In one place, Google says my all-time views are 313,846…and in another Google says it’s 474,236.  How can the same source (Google) give such wildly different numbers for the same thing?

Are they the same number…?

Okay, so the first thing I think is that they measure two different things.  I dig into the help files.

The total views number (313k), according to the Google My Business help files, includes how many times I’ve appeared on:

• Google Maps
• Google Search
• Maps for mobile
• Google+ page (as in, my business “profile”)
• Other people’s streams (my individual Google+ posts…but it doesn’t count ones that might show up in Gmail)

It also counts views of photos I’ve posted (which might show up in Google+, Search, Image Search, Maps, and other locations).

Okay, and the other, larger number?  It’s supposedly just views of my profile and its contents.  So that would still be my G+ posts, page views, and photos, but probably not Maps, Search, and so forth.  This measure seems to be just looking at views within Google+ itself.  But it’s the bigger number at 474k.

How can it be that looking at a subset of locations returns a bigger number?  As I write this, I have to think it has to do with the time frame.

According to the help files, the smaller number covers the time period from October 1st, 2012 to the present.  I can’t find any data on when the “classic G+” profile number started counting.  However, Google first announced business pages on the social site on 7 Nov 2011.  I created my business page at some point between then and 17 Dec, 2011, when CircleCount.com added the page (9 followers!)

In other words, it might be that the profile number is larger because it started counting almost a year earlier…a year in which Google+ was new, and when business pages as a feature was new, and people were trying everything out and seeing what was available.  Then as the looky-loos stopped visiting and the novelty of business pages faded, the Google My Business count started.

At least, that’s my best guess.

Back to the analysis…

The real takeaway here is that analysis isn’t always a clear-cut world where every number is accurate and describes what you think it describes.  And that’s why looking at your website numbers isn’t straightforward and makes analysis a complicated endeavor.  Many measurements are mildly murky.  There’s a whole layer of knowing where inaccuracies might be coming from.  Or watching out for inconsistencies from one source to another (or in this case, from the same source).

Sometimes I’ll send out things you can check for on your own website in my monthly mailing list.  For example, I recently talked about how you can see if spammers have gotten some dubious data into your Google Analytics.  In fact, “data hygiene” is its own area of work, where errors and duplicates and other data problems get cleaned up.  So there are some simple, basic things you can do.  And then there’s the other stuff.

Of course, that’s what I do.  So if this sounds like something you don’t want to do, or seems stressful, or brings up questions in your mind, get in contact with me and maybe I can help out.

Have you run into dubious data or murky measurements?  What impact did it have?  How did you deal with it?  Let me know in the comments!

Image credit:  Wikimedia Commons

Note:  Google says it will soon be defaulting to the “new” version of Google+ for all users, so who knows how long the “classic” version will be available.  

Data & Trends

Trick-or-treet (ToT) visitors by hour for the most recent season by hour, with previous 4 years for context and year-over-year comparison:

Costumed visitors for competitor businesses:

• Observation 1: 5
• Observation 2: 2
• Observation 3: 13
• Observation 4: 6

What Does It Mean?

This year’s event lands squarely within the first standard deviation of the last five years. Of course, it comes nowhere near the numbers of 10 years ago (e.g., 2004: 52 visitors), but a new lower pattern has been established. Relevant features include:

• No visitors prior to 6pm; earlier periods averaged 0.8 visitors before 6:00.
• Continuation of pattern of no visitors after 9pm.
• Establishment of a later peak time. Most visitors during 7:00 hour; a decade ago 73% of visitors arrived during the 6:00 hour.

The new dynamic can be traced back to the launch of competitor events such as this year’s Hunger Goblin Trick Or Treat, and Halloween on Holman. Increased competition from small business core areas in the neighborhood is meeting a majority of, but not all, demand.

What Should We Do?

Recommendation 1: A/B Test Better Candy
Competitor businesses were giving “fun size” and “bite size” selections of consumer-grade product. Test larger and higher quality options, using capturing at least metrics of visitor satisfaction rating and active word of mouth referrals (via exit survey question) as well as increased returning visitor percentage.

Recommendation 2: Reduce Hours
Given the reduced hours of visitors, focus on peak periods by only turning on the porch light between 6:00pm and 9:00pm. This reduces costs associated with Halloween overhead.

Additional Information

Costumed visitor metric based on photographic data taken between approximately 2:25 and 2:30pm on 31 Oct 2015 on the corner of the major intersection serving a main commercial street for the neighborhood. Costumes counted if waiting to cross or actively crossing the street.

An additional benefit to Recommendation 1 may also solve the identified problem of “too much damned candy left over.” While purchasing statistically-appropriate amounts is a direct way to address the problem, higher quality remainder product may result in a reduced perception of the leftovers as a problem in the first place…an indirect solution to be sure, but one to keep in mind during strategic planning for next year’s event.

I recently discovered a new show that I’m really enjoying. Well, new to me–the show is in its third season. It’s called Start Up.  Host Gary Bredow travels the United States talking to the owners of various start up companies. It’s also a bit ‘meta’ because the show itself is one of the first products of Bredow’s own start up company. The businesses he visits all seem to be in the first couple of years of being open, having achieved at least a moderate level of attention and success, but not necessarily far enough in that they’re operating reliably at a profit.

The companies are in a wide range of industries as well. The episode I watched last week included an aerial dance/circus arts studio, and an the makers of a electronics-filled teddy bears that teach diabetic children how to manage their health. Bredow features two businesses per 30-minute episode, interviewing the owners about their challenges, their process in getting started, how they financed the start up, where their ideas came from, and more.

Having been a business couneslor for a program of the U.S. Small Business Administration for several years, I can say that their stories seem pretty typical, though very much weighted toward the success end of the spectrum. (It wouldn’t make very good TV to have a series in which 60-75% of the featured business owners talk about how their great idea sputtered and unceremoniously died because of poor preparation or lack of demand.) But the variety of where ideas come from, how money gets scraped together, and what kinds of unexpected pitfalls crop up along the way are all very familiar.

The show also drops in a few pieces where one business expert or another gives some tips or perspective related to the start up process.  And each profile ends with Bredow summarizing the interview in a pun-filled wrap up.

I have been enjoying it, but I knew I had to put up a brief piece about the show here on the blog when he asked Dori Ross, a maple-products maker in Vermont, “Social media–how has that impacted your business?” She tells him that she’s on Facebook, Twitter, Instagram, and Pinterest, and that she has two blogs. That’s all fine (though it’s a lot to do well). But then she stabbed me through the heart. She said, “And I don’t know if it works.”

It’s tragic, but it’s common. People hear that social media, or a blog, is this powerful thing that you can do for your business. They jump in. They do a ton of work to try and figure out how to work these new tools. They post pictures. They comment, and share. They write articles. And they never look to see if it’s actually helping them. It’s just blind faith that this stuff works, and in many cases it doesn’t.

I would bet that Dori could probably drop half the social media or blogging work that she’s doing. I don’t know that for certain, or which things she could drop, because I’m not privy to her data. Heck, I don’t even know if she’s collecting data.

Digital strategy is, in large part, determining what doesn’t work and dropping it. It’s choosing to not do things that are less effective so that you have more time and effort to do well the things that do work.

I beg of you, talk to me if Dori’s story sounds familiar. It is absolutely possible to figure this kind of thing out. It’s possible to figure out which social media sites send you the most traffic. Or send you the most buyers. (It’s not uncommon for visitor volume and most likely buyers to come from different places.) Let me help you figure out a few things you can drop from your To Do list. Because I can almost guarantee you’re doing things that don’t contribute to your success.

Or if you’re not doing much of anything online, maybe it’s time to add one highly targeted thing to your list.  I could help you figure that out, too.

If you’d like to watch Start Up, they have a utility on their website to look up the TV schedule in your area.

Shareaholic’s recent 3rd Quarter Social Media Traffic Report has me thinking about key performance indicators and what my clients really want from the Internet.  There was a very interesting discussion between a number of internet-marketing type people related to the report over on a Google+ post by Ana Hoffman of Traffic Generation Cafe.

While I want to get back to that discussion and what it has to do with you in just a moment, I first would like to point out that while I kind of criticized their 2nd Quarter report in a previous blog post, and might say a couple of things that seem critical in this one, I harbor no ill will toward Shareaholic.  My purpose is to go a little deeper and not take their data at face value, as I see some people doing.

This might go without saying, but you are not the average Shareaholic website, and by extension, your clients are not average either.  But I’ll get to that in more depth after we cover what you want the Internet to do for your business.

Okay, so let’s look at teasing out the bits that are useful so we can avoid being led astray by data that doesn’t apply (even if lots of folks online are telling you that broad averages are somehow equal to the reality of your specific business).

 What are Key Performance Indicators (KPI)?

Key performance indicators, or more frequently referred to as KPIs, are measurements of success.  Where many business owners miss the mark is ignoring the first word: key.  In other words, the indicators that are key to success.  Lots and lots of things can be measured.  But the question that doesn’t get asked enough is, “Is this indicator critical to measuring success?”

Why is this important?  Why do I often limit clients to just 3 (or maybe 4) KPIs?  Because there’s just too damned much information out there and you’ll get overwhelmed looking at everything.  So you first need to figure out what is most critically important.

How Do I Get More Traffic To My Website?

That’s a great question. It’s an important question. But is it the most critical one?  Here we go back to Shareaholic and Ana Hoffman’s post.  Shareaholic’s Danny Wong posted on their blog some of the data in an article entitled “In Q3, Facebook Drove 4x More Traffic Than Pinterest [REPORT]”

“OMG!  Quadruple the traffic?  We need to be on Facebook NOW!”  Yes, I hear you thinking that.  But what are we talking about here?  KEY performance indicators.  Is the number of people who come to your website the most important thing?  Maybe not.

On Hoffman’s post, different people brought up a bunch of “yes, but…” scenarios, detailing things they thought were more important than simply the number of visitors.  These include:

• Engagement level: the likelihood that someone will interact with your brand by endorsing (Like, +1), sharing, commenting, replying, etc.
• Conversion rate: the likelihood that a visitor will subscribe, or purchase
• Return on Investment (ROI):  Mentioned by Thomas E. Hanna in relation to bloggers, this could refer to either time invested, or money
• Audience: Are the right people–your target customers–the ones who are coming?
• Shares/endorsements from your website via a social-share button:  Visitors already on your site giving your social media exposure
• Shares that spread your brand page content through social media: Social media exposure that might not mean people going to your website
• Connecting with previously-established “fans” (and whatever value there is in those connections)
• Connecting with new “fans” (and the value that those people have)

Now that you’ve seen some of those other options, how convinced are you that the number of people who visit your website is the Key Performance Indicator for you?  If you were given this list and told you could only look at one number, what would you choose?

Ardea’s 3Q 2014 Social Media Traffic Report

Okay, now back to that other point.  You are not an average business, and your customers are not average people.  To illustrate why you might not want to take aggregated reports like the Shareaholic numbers at face value, I’ve done my own version.  What they did (at least for the bit mentioned in the title of the blog post) was look at how much traffic came to all of the websites that used their service, and see what percent of that traffic came from each of the “top 8” social media sites.  Using their data from September 2014, the top sites look like this:

1. Facebook (22.36%)
2. Pinterest (5.52%)
3. Twitter (0.88%)
4. StumbleUpon (0.41%)
5. Reddit (0.18%)
6. Google+ (0.07%)
7. YouTube (0.04%)
8. LinkedIn (0.04%)

Now I’m going to do the same.  I’m taking the visits from sites I run, and the sites of my clients for whom I have Google Analytics data (for reasons I won’t go into here, it’s difficult to properly mix-and-match data sources, so I’m trying to keep the data clean).  Using my own set of “websites that used my service” I’ll do the same calculation–what percentage of total traffic came from each of these 8 social media sites in September 2014.  My rankings are as follows:

1. Facebook (0.42%)
2. StumbleUpon (0.31%)
3. Twitter (0.15%)
4. Google+ (0.09%)
5. Reddit (0.08%)
6. Pinterest (0.03%)
7. LinkedIn (0.02%)
8. YouTube (0.01%)

I should point out that my clients tend to be very small businesses, many of whom haven’t been on social media for very long.  But notice that Facebook doesn’t even account for half a percent of traffic.  It’s still the top position, though.  Pinterest sank from #2 to #6, while Google+ rose from #6 to #4.

In addition to the shifts in rank and in total percentage, my numbers showed WordPress at 0.08%–as much as Reddit and nearly as much as Google+.  Blogger beat the bottom three with 0.05%, and Yelp matched Pinterest.  So for my clients, the “top 8” social sites are a different set of social media sites than they are for Shareaholic’s clients.

If I do that “grading on the curve” technique of dropping the highest and lowest traffic sites, the rankings change again:

• StumbleUpon (1.8%),
• Twitter (1.0%),
• Google+, Reddit, and Facebook (tied at 0.7%)
• WordPress (0.6%),
• LinkedIn and Blogger (tied at 0.3%)
• Pinterest and YouTube (tied at no traffic whatsoever).

To belabor my point a little: if you see a big report on what the best sites are for a business to market on, take it with a grain of salt.  Don’t assume that a large collection of websites will return the same results as your website will, particularly if the sites aren’t of the same size or industry as yours.

And to bring this around to the larger issue, what you need to do as an entrepreneur or the owner of a small business or startup, is to zero in on where these two things intersect.  Once you determine your key performance indicators (and make sure they’re the “critical few” as author and analyst Avinash Kaushik puts it)

A Tiny Bit of Criticism

As MaAnna Stephenson notes, the Shareaholic data is pulled from Shareaholic users, which may have a certain bias toward brands that have used Facebook heavily for some time and therefore weight in favor of Facebook.  Ana Hoffman and Danny Wong both point out that the data includes paid traffic.  Since Facebook ads happen on Facebook, but Google ads do not appear on Google Plus, that also weights the Shareaholic results in favor of Facebook.  In essence, Facebook gets credit for organic traffic and paid traffic, while Google Plus only gets credit for organic traffic but not paid.  So it’s not really comparing oranges to oranges; it’s more like fruit salad to oranges and saying the fruit salad is better because you get more types of fruit.

Shout-outs

I’ve already mentioned several people above who were involved in the discussion, but those that I didn’t mention by name yet and contributed to the list of possible Key Performance Indicators you might want for measuring your online success are Eli Fennel,  Thomas E. Hanna (whose “Weekly Insider Goodies” list from BlogPhoto.tv provided the image for this blog post), Justin Chung, and Kymmberly Gail.  Thanks for a great conversation.

Testing is something everyone with a website should be doing, but almost nobody does.   It’s the crucial activity to take the guesswork out of your site, whether you’re a blogger, a small business, or a nonprofit organization.  It impacts your written copy, your design…everything. I believe this so strongly, my professional recommendation is that you slap together a non-embarrassing website for cheap (more on this in another post), and then start testing different options.  Don’t hire a photographer or a web designer, or a developer until you’ve done your testing to discover what works for your audience.  Then you can tell them what you actually need.

Why should I be testing on my site?

Here’s why:  You’re just not going to guess correctly on a regular enough basis to be useful.  Professional designers and developers will probably be able to implement things that are more likely to get it right, but still not often enough.  People basically suck at guessing what’s going to work.  I’m talking to you.  And I’m talking about myself.

An Example: Testing vs. Guessing

To prove my point, here’s a little quiz: Let’s say you have a registration form you want people to fill out.  You wonder if adding some text about privacy will improve registrations.  Do you:

1. Add “100% Privacy — we will never spam you!”
2. Add “We guarantee 100% privacy. Your information will not be shared.”
3. Leave it alone — nobody’s complained about it, so don’t mess with what isn’t broken.

Give yourself a moment to choose what you would do.  This process is how most companies go about making decisions about their website.  They come up with some ideas and pick one.  In this case, these options are actually from a real series of tests that were written up as a case study on Unbounce.com, meaning you can find out the “right” answer without doing the test.

But first you have to guess.  There’s a 33% chance of getting it right randomly (even less so if you were to brainstorm more options).

Now for the data

For starters, the third option is what the case study already had in place, so it was the control.  No change in what you’ve showing means no change in your results.  If you chose the first option, you have done yourself a disservice.  That change reduced registrations by almost 19%.  But the second option on the list (the one that doesn’t mention spam) actually increased registrations by nearly 20% compared to the control.

So should you add a comment about privacy to your form?  Maybe…and maybe not. You won’t know for sure unless you do these statistical tests and see what impact they have.  I’ve done enough tests (like the coupon testing I talked about in a previous post) and read enough case studies like the one this example is from to know that sometimes big changes make no difference at all, and tiny changes can have a huge impact.  I saw one example where simply changing the color of the background, or a single word on a button, or the number of bullets in a list either boosted the goals of the page (or made them plummet!)

Test first.  Get the real answers.

THEN go out and tell your developers and designers what they need to do.  Because, with data in hand, you’ll know.  You’ll have eliminated the guesswork.

You do your best to look good, to be professional, and to get the job done right.  But sometimes you just mess up.  It’s human.  And sometimes, it’s a little (or a lot) embarrassing.

Luckily, a regular review of your website data can help catch some of those mistakes before too many people notice.  Here, for your viewing pleasure, are a few of the errors I’ve caught while swimming in the data stream.  The identities of those at fault, and the businesses they worked for, have been anonymized to protect their dignity.  Because really, wouldn’t we all like to be seen as a little better than “merely human”?

I’ll refer to all of the websites as “mysite.com” and whoever is in charge at the company as “Mr. Owner”.

Web Data Revealed These Goofs

• I’m Sending Away Visitors I Already Have…to Myself!  Looking at the data from this site, I noticed a strange pattern.  Lots of visitors were leaving mysite.com—abandoning it to go instead to mysite.com. Also, the #1 source of incoming traffic to mysite.com was the site mysite.com.  Luckily for Mr. Owner, I knew immediately what was the problem at set out to fix it.  These so-called “self referrals” are usually due to incorrectly installed tracking code.  Sure enough, after a little exploration, I found some pages that didn’t have the code on it, so if a visitor went to that page, it was like they vanished (at least from the point of view of the tracking software).  And then they came back to a page with the code on it, the tracking program thought, “Hmmm.  Where’d this new visitor come from?  Looks like it was mysite.com!”  Oops.
• It’s a Miracle!  I’m Making Sales Without Even Having A Store!  In this case, Mr. Owner had more than one site.  One had an e-commerce site as well as some other sections.  Another website was only a blog.  But the person who had installed the tracking codes had gotten things mixed up and installed the blog-only account number on the blog…but also in the store section of the other website.  From the dashboard, then, it looked like his store was making no sales (despite the fact that he was getting orders), but someone people were buying things directly from his blog.  Crazy!  I made sure all the pages on both sites had the right account number, and now everything looks like it should.
• You Know What’s Better Than What’s In Your Cart?  Knockoff Drugs From Overseas Somewhere!  Mr. Owner was hacked!  But it was a secret hack.  All that happened was that every once in a while (not every time, mind you), a person might randomly be sent to a drug-peddling site overseas when they clicked the “checkout” button in the store.  The issue was revealed, in part, because it looked like a strangely high number of people were supposedly clicking a link that we couldn’t find on the site, and leaving for a sketchy-looking website.  We had the site’s web host do a security check and that discovered the rest.
• Button? What Button? Oh, You Didn’t Want To Sign Up, Did You?   Yeah, speaking of clicking on a button, if you have a page that says “Just fill in this brief form and click ‘Sign Up’!” you should probably have a button that says ‘Sign Up”… or a button.  Mr. Owner felt very embarrassed about this one.  But he put it in the middle of this list in the hopes that readers don’t notice (but if you tried to sign up for the Ardea Coaching mailing list in the last week or so, there’s a button now).  The data that revealed this problem?  I’d set up an alert to specifically track button clicks as a goal conversion and I was alerted that there had been no conversions in over a week.  So hurray it wasn’t longer than that!
• We’ll Provide Your Professional Services!  And a Russian Mail Order Bride, Too.  The last site I’ll talk about had indicators similar to the knockoff drugs issue.  There were an odd number of exits following a link.  And that link happened to go to a site that… well, it was one that would not be good for your boss to see you looking at in the office.   But in this case, the site hadn’t been hacked.  What had happened was that the person who had designed the website originally had included a link back to their site.  Over time, they must have let the domain name expire (or sold it).  Whatever happened, the domain changed hands from a website template designer to a mail order bride site.  Every page on this professional services website had a link on it to a somewhat naughty website.  That wasn’t the kind of “professional services” they were selling, either.  I quickly removed that link from the footer so nobody would question the integrity of the site owner.

What Do You Do?

While ‘saving clients from embarrassment’ isn’t what I usually say when someone asks what I do, that’s sometimes a side effect.  And a reason you should always review the data you’ve got coming in from your site because you never know what surprises you might find on your own site.

Image source: https://www.flickr.com/photos/hslphotosync/5939459500/ (no modifications except resizing)

Testing my 1924 Underwood 3-Bank Portable typewriter

In this post, I’m going to talk about testing.  In particular, an example of low-tech testing that I really did for a former employer and some of the non-intuitive things we learned without investing in a ton of technology.

Right now, you might have some part of your business that you suspect could work more effectively.  How do you know if you should make a change or not?  Or maybe you know something must change, but you don’t know what the new thing should look like.  How do you choose?

The short answer is: data.   You want to test different options in the real world in a way that will allow you to see what option works best before committing to one.

High Tech and Low Tech Testing

At a basic level, all testing works basically the same.  You have at least two options and you measure how the options do against one another.  Low tech testing of marketing has been going on for at least a century (depending on what kind of analysis you’re looking at).  In the 1920s, for example, a number of studies were published about what coupons printed in newspapers and magazines had the highest response rates.

Over time, of course, technology has increased our ability to fine tune what we learn.  In the 1940s, magazine publishers were able to more easily do “tip-ins” (ads or features or whatever that are added to a magazine, usually after it’s been bound).  This opened the door for tipping in different versions of an ad to see which sold better.

In the era of the Internet, back in the early days of Amazon.com, you used to be able to hit the refresh button on your browser and see different front-page layouts.  You could actually reveal the different test versions they were doing at any given time.  Now, however, that testing has gotten even more subtle.  But it’s still all the same idea.  Does Version A perform better than Version B?  Let’s try them both on real users and see!

Back to Basics: My Mail Order Coupon Tests

As some people know, I’m something of a tea geek.  As such, I’ve worked for a number of different tea businesses (as well as having my own educational/consulting company related to tea).  In one of the businesses I worked for, we noticed that we had quite a lot of first-time mail order customers, but a very low percentage of repeat business.  It seemed like a reasonable goal to increase repeat business.  The way we decided to do that was to add a coupon in every outgoing mail order package.  The more coupons that got redeemed, by definition the more repeat orders we were getting, since the only way to get a coupon was to have already ordered at least once before.

We set up a baseline by sending out coupons for a month or two.  Our main metric was essentially the percent of total orders in a month that used a coupon.  We could have looked at how many orders were repeat orders, since at least theoretically, the coupon could have reminded people to be repeat orders without them actually using the coupon.  As a measure of effectiveness of the coupon as a repeat-order-building tool, a straight percentage was going to be accurate enough and easier to calculate given our order system.

Once that baseline had been established, we started testing.  Each month, I would design two coupons.  Version A was the ‘control’.  That is, whatever version had been performing the best so far.  It was our standard.  I’d also make a Version B that changed something about the coupon to see if that change increased the redemption rate.  I’d make an equal number of copies of each coupon, collate them so they alternated in the stack, and gave the stack to the warehouse folks.  That way as each order went out, the first one would get Version A, then Version B, then Version A, and so forth.  For tracking purposes, each version got a slightly different coupon code to use when redeeming so we knew which one the customer had gotten.

What We Tested

Over the course of about two years, we tested lots of things.  We tested the color of the paper it was printed on.  We tested amount of the discount.  We tested a discount described as a dollar amount (“$10 off your next order”) and a percentage (“10% off your next order”).  We tested serif vs. sanserif typeface.  We tested location of different parts, such as whether the discount information put at the top so it was the first thing made a difference compared to having it in the middle of the coupon after some kind of headline text.  Font size.  Line weight.  Wording.  All kinds of variations on all kinds of things.

The Results

After each pair of coupons expired, I’d do the math to figure out if we had a winner.  Often, we didn’t.  Statistically speaking, although one might have been a little ahead of the other in terms of raw numbers, it wasn’t enough of a difference.  It’s what’s called the “null hypothesis.”  You assume any difference is due to chance or inaccuracy of the numbers or whatever unless the difference is “statistically significant.”  We had lots of null-hypothesis results.  And that was important.  On the one hand, we’d just spent a bunch of time testing and found, essentially, nothing.  On the other, we could safely use either option knowing it wouldn’t hurt.  If the boss really liked Version B much better than Version A?  Great.  Doesn’t matter.  Go for it.  We’ve essentially got proof that it’s not worth worrying about any more.

Some of these “no result” tests were still instructive, however.  A 10% off coupon pulled just as many responses as one for 40% off (which, if I recall correctly, was the highest amount we tested).  Why use a larger discount if it didn’t get you any more purchases?  We found no significant difference between percent-off and dollars-off.  But in that case it was a little trickier.  The percent-off coupons resulted in larger purchases.  They didn’t result in more frequent purchases which is really what we were looking for, but the average order size was much larger, so when I noticed that, we stuck to percentages.

And there were also some surprising results that did show a difference.  An expiration date 2 months in the future didn’t do nearly as well as one with an expiration date 3 months in the future.

One big result was a “positive message” vs. “negative message” test.  One version was something like “Thank you for your order!  We appreciate your business and would like to offer you this discount on your next order to show how much we love our customers!”  Something like that.  Gratitude, customer-focused, that kind of good stuff.  Version B was totally playing off fear and loss.  It went along the lines of “You don’t want to find yourself without tea, do you?  You’d better use this coupon to reorder early or you might run out!”  Being negative soundly defeated the nicer version.

But the biggest, most decisive victory between versions was perhaps the least intuitive.  All of our coupons for all of the tests were printed on quarter sheets (4.25″ x 5.5″).  One round, I used no border.  On the other, I used a dotted-line border around the outside edge.  The one with the border showed a huge increase in redemption rate.  Like not just 5% or 10% more, but more on the range of multiple times as many.  Following that, I did lots of testing of the border, but I’d apparently hit the right one the first time.  A double solid line or a fancy scroll border or any other version didn’t make a difference.  But a dotted line around the outside was our golden ticket.  Why?  Maybe because it reminded people of a coupon you had to cut out of a magazine?  I don’t know.  They just needed to enter the coupon code on our website, or tell it to the person on the phone, so no cutting out needed to happen.  But for whatever reason, we found success in an otherwise completely irrelevant dotted line.

What This Means To You

Think back to that thing you’re not sure what to do in your business.  Apply some testing.  There’s lots of online technology to help you do that, or you may be able to devise something low-tech to track, like our two coupons with different coupon codes.  But do start thinking of making decisions based on data.  You may reveal your own “dotted line”—that non-intuitive thing that for whatever strange reason, makes your customers more likely to buy and buy again.

A couple of weeks ago, Shareaholic published its social media traffic report for the second quarter of 2014.  In it they looked at social media data from 8 of the more well-known sites:  Facebook, Pinterest, Twitter, StumbleUpon, Reddit, YouTube, Google+, and LinkedIn.  Based on their social referral data, which come from over 300,000 websites, Facebook drives the most traffic to websites, hands down.  Like nearly 25% of all visits.  You might then immediately jump to the conclusion that your business needs to be on Facebook because it’s the best.

Let’s assume the numbers are correct and accurate.  It’s not something I always assume, but in this case I want to thinking about what they mean.  To do that, we have to think about the situation that might have contributed to those numbers.  How do people on those social media sites end up on the various websites that they measured?  In other words, what is the pathway that is being measured?

A Visitor’s Path

The most likely way, of course, is the social share.  Someone shares a link to one of the websites in a social media post.  You know, probably with a comment like, “OMG. This is so cool, you have to see it!” or “Great article” or whatever.  Their friends/followers see the post, click on the link, and *ding*!  They’ve been tracked as referral traffic from a social media site.

So far, so good.  The unsuspecting business owner still says, “Great.  If I’m on Facebook, more people are there, more people will see the post, and more people will click on the link and visit my site!”  Eh.  Not so fast.

The Other Half

If we apply systems thinking concepts to this, we can recognize that we’re only looking at the second half of the full system.  At this point we haven’t looked at how and what gets shared, and that’s where some issues start to be revealed.

Many sites include social sharing buttons.  You know, the little buttons with all the social media sites where you can just hit it to create a new post with the link to that article already filled in.  A person reading an article on a site might copy the web address and paste it into their own post manually.  But many people will use those buttons to share.  And here’s where we start really thinking critically: how much influence do those buttons have on the referral traffic back to the site?

I’ve been on many sites that only have a Facebook and a Twitter button.  I know that sometimes I want to share the article on Google Plus, and have in a few cases, not shared the article because there was not a G+ button.  As a result, my G+ followers didn’t see a link to the page, meaning they didn’t follow it and become a social-referral visitor.  Meanwhile, Facebook and Twitter are getting higher amounts of exposure because of the buttons, thereby increasing the social shares on those sites.

So here’s the thing:  Is Facebook really driving more traffic, or are websites just more likely to have a “Share on Facebook” button?

We could even go further with this…for example, some social share systems have a place where you can click and reveal a button for all the sites you’ve never even heard of.  But data shows the more clicks you need to do, the fewer people will follow through.  And people are more likely to click the first item than the second one, and so forth.  So even the order of the buttons might make a difference in how and where an article is shared.

The Answer: Look at Your Social Referral Data

The answer to this confusion brings us back to the image on this post:  test, test, test!  The only way to really be sure for your website is to look at your own visitor data.  Just now, I checked the last two months of data for ardeacoaching.com and my other site, teageek.net.  One site got a strangely-round number of 60.0% of social media referral traffic from Facebook.  The other got 9.86% of its social referral traffic from Facebook.

Looking at Shareaholic’s article, you might be tempted to think you have to be on Facebook.  In fact, they say in a bold header, quite definitively, “Facebook is, by an extremely wide margin, the #1 source of social referrals” But I’ve got a site where over 90% of social referrals come from not Facebook.  It gets roughly three times as much traffic from StumbleUpon than Facebook, but from Shareaholic’s standpoint, Stumble upon only drives a measly 0.6% of traffic.

Who am I to believe?  Well, my own numbers, of course.  It doesn’t matter what the average is.  It matters what is working for you.  And if you don’t like what you’re getting, experiment with new things and see if you can improve your results.  That’s why systematic experimentation and testing is usually part of my digital strategy recommendations.  You can’t tell what results you’ll get by looking at averages, and you can’t tell what will work best unless you try different things and measure your outcomes.

Which sites work best for you?  (Or, if you’re not sure, what challenges do you face finding out?)  Leave your answer in the comments.

And don’t forget to share this article on social media by using one of the buttons!

Image source: my own notebook where a friend and I were testing different fountain pens and different inks.  Not social referral data, perhaps, but testing nonetheless.