Split-testing is one of the basics of marketing and the support for this within the AdWords system is well implemented. For those that are just learning about the feature it is a process in the system to test your ads by splitting the traffic between two or more ads to figure out which ad produces the best result.
While this might sound like a simple process it certainly is not and the devil is in the details. The set-up and mechanics of a split test are very easy. The challenge is in the analysis of the results.
Decisions Need Data
Making a decision on a split test requires a meaningful level of data which is the first challenge you face in a split test. On one side you want to make a decision as soon as possible but on the other hand you also want to make a good decision. So what is meaningful data? Great question and one without a simple answer and it depends on what you are trying to test. If the goal is to test the click through rate that can be quick and cheap but that is not meaningful to most businesses. The CTR is important to Google because it is how they convert impressions to revenue but for an advertiser it is not that important. Advertisers need to look to where value is created in their business and normally this is with a conversion.
What is a Conversion?
A conversion is simply an action that creates value for the business. Common conversions are orders, leads, subscriptions, or reading of specific copy but they can be any measurable action the business considers of value. As you step down through these the rates drop because of the level of commitment required. In most cases you will get more leads than direct orders and you will get more free subscriptions than specific leads.
Minimum Data to Test
For example, let’s assume the conversion is a sales lead and that we have a 2% conversion rate. So we get one conversion for 50 clicks and 50 impressions for each click so it takes 2,500 impressions to get one planned conversion. Now the question becomes how many conversions do you need to get a number you can trust to repeat the performance. There are lots of fancy ways to do this math but our experience has taught us that if we have four events in a monthly period the results are very likely to repeat and we call this the Rule of 4. Simply stated, this rule says that if you get the result you expected four times in a row it will probably continue to give you those results. For the stats types that are flipping coins this is four heads in a row and it probably means you are flipping a double headed coin. The Rule of 4 gets you to about 80% probability in most cases. This means the keyword needs to have 10,000 impressions to be a reasonable test.
What’s Your “Margin of Error?”
You hear this all the time on the news and it applies to your market testing. To test your margin of error create a split test with a duplicate ad and run them. What you will find is that results in marketing vary even when everything is identical and this test will help you understand the range of this error. We have done this hundreds of times and our experience has shown us that on a CTR or Conversion rate that is in the 2% range we will have a margin of error of +/- .2%. In other words a result from 1.8% to 2.2% is all within the margin of error and your result has to be outside this before it can be counted as a clue of what is going on. The farther out the margin of error the greater your faith in your data can be.
Clues Are NOT Facts!
We all want to deal with facts but the reality is that all marketing data are clues. The more clues you have the more likely they will direct you to the proper decision but they are still only clues. To get from a fact to a cause is difficult and to get from a clue to a cause is even tougher. This is where experience with the account really helps because you will get clues from multiple areas and associating them and seeking patterns that work are the Art of AdWords.
The Bottom Line
Test when you can and trust your gut when you cannot.