Brad Geddes / PPC Geek
Official Google Ads Seminar Leader.
Author of Advanced Google AdWords.
Co-Founder, Adalysis.
(312) 884-9017
Brad Geddes's Theories on Marketing Ad Testing: Are You Using the Wrong Success Metrics?

Ad Testing: Are You Using the Wrong Success Metrics?

Everyone knows you should test ads – that’s not debatable anymore. However, what is debatable is which of the ads you are testing is actually the best ad.

I recently polled a large group of marketers and asked them of these ads, which one was the best.

image

Take a moment to decide which you think is the winner.

Here were the responses:

image

7.3% thought there wasn’t enough data to determine a winner. Those are the ones who are correct.

The Current Metrics Are Flawed

When you consider all the stats needed to determine a winner; you need to work from these numbers:

  • Impressions
  • Clicks
    • CPC
    • Cost
  • CTR
  • Conversions
    • Average sale amount

From those metrics, the most commonly generated stats to determine a winner are:

  • Conversion rate
  • CPA (cost per acquisition)
  • ROAS / ROI
  • CPI (conversion per impression)

Each one of those metrics has a fundamental flaw.

Conversion rate does not take into account impressions, click through rate, or average sale amount.

Cost per acquisition does not take into account volume (impressions and clicks) or average sale amount.

ROI / ROAS does not account for volume (impressions and clicks).

Conversion per impression does not take into account average sale amount.

It Starts at the Impression

Assuming your campaign is targeted towards sales (we’re going to leave the branding discussion for another day); you should be measuring from the impression.

You choose a keyword. Someone searched for your keyword. That combination creates an opportunity for your ad to be displayed. It also creates an opportunity for a conversion.

Yes, every single impression has an opportunity to generate a sale. Therefore, your measurements should start at the impression as well.

Clicks Through Rate vs Conversion Rate

One of the problems with testing is that you have two big metrics you’re often examining: CTR and conversion rates. However, which of these scenarios is better?

  • High CTR, low conversion rate
  • Low conversion rate, high CTR
  • Mid range CTR, mid range conversion rate

It’s an impossible scenario to measure unless you are using a metric that incorporates CTR, costs, conversions, and average sale amounts into a single number. That’s what Profit per Impression will do.

Profit Per Impression

I first started using this metric in 2002 and first started writing publicly about it in 2006. I’m still amazed at how little adoption this metric has since it is the overall best metric to use for testing and its very easy to calculate.

To calculate this metric, all you need to do is take the original stats from above and add a few pieces of data:

  • Revenue (this might take some effort if you are doing ecommerce with variable sales amounts and product costs, but it is worth the effort)
  • Profits (simple: Revenue – cost. I’m ignoring the complex aspects here of adding lighting, salaries, etc)

With those metrics, calculating PPI (profit per impression) is easy; just divide profit by impression:

image

It turns out the ‘best’ ad is not the one with the lowest CPA, the highest CTR, or the highest conversion rate.

It turns out that 4.2% of the original voters were correct (or 11.7% if you count those who voted that there wasn’t enough data); but since they didn’t have the correct data it was most likely due to a good guess.

To prove the PPI theory, let’s see the extrapolated stats as if every ad actually received the same number of impressions:

image

As you can clearly see, ad 1 is a clear winner in terms of total profit.

If you happen to be wondering why these stats seem so high on the revenue basis; its because the revenue is not based upon a one time sale. It’s the lifetime value of the customer. Always take into account lifetime values. Here’s the stats if you base the numbers off of just first month revenue:

image

While ad 1 is still the clear winner; from a single month basis the highest CTR ad is actually losing money and doesn’t appear to be profitable. To get a clear picture of your actual revenues, if customers buy from you more than once a year, which includes subscription products, then you need to calculate PPI from the entire customer’s revenue and not just the first time sale.

The Quality Score Wrinkle

There are some times when I won’t actually pick the best PPI ad; and that is often due to Quality Score. If the second best PPI ad has a much higher CTR than the best PPI ad, then I will often pick the 2nd best. Because CTR is such a huge quality score factor, if you pick an ad with a much lower overall CTR: you could cause your Quality Score to drop which in turns means you average position will decrease or your average CPC will increase (or both).

If in this test, ad 4 (with a CTR almost double all the other ads) didn’t have a $0.39 Profit per Impression, but was instead in the $0.43-$0.45 range; then I would have declared it the winner instead due to the Quality Score wild card. Or, I would have set up an ACE (AdWords Campaign Experiments) test between the two ads to see if I could get some Quality Score insights between the two ad and keyword combinations. However, it was enough lower in total profits that its not the winner by a large margin; and therefore, I would pick ad 1 as my true winner.

Conclusion

Testing is easy. Pick at least one ad group and:

  • Write a few ads
  • Wait
  • Measure the results
  • Learn from the losers
  • Delete the losers
  • Write another ad or two
  • Hit save
  • Wait
  • Measure
  • Repeat

There’s nothing difficult about testing. However, if you are using the wrong success metrics in your test; then your time and efforts will be wasted. By using Profit Per Impression metrics, you can be assured that you are pick the ad in your tests that will bring you the most profit for your paid search account.

No Comments

  1. kevinjgallagher
    February 20, 2013 at 3:27 am · Reply

    I think the reason a lot of time this is not implemented is because on a lot of smaller campaigns the client usually doesn’t give you the profit or knows the lifetime value.

    Yes, and I know you would say these are metrics that you should know and I agree. It’s getting them that’s the hard part. Not only this getting agencies to understand this also when they sell these services can also be difficult.

  2. brad
    February 20, 2013 at 7:04 am · Reply

    Hi @kevingallager – I totally agree. The math is the easy part; figuring out the information or getting it from your customers is by far the hardest part of this entire equation.

  3. poxc2006
    February 21, 2013 at 8:50 am · Reply

    I want to utilize this more often. The difficulty I’ve found in practice is the workflow. When you have many many ad groups, I’ve found it to be time intensive to export them to excel one-by-one (perhpas there’s different way I’m all ears). So often if I’m swamped it falls by the wayside. I would love it if AdWords integrated the metric into their interface, or into AW Editor, instead of needing to do so much import/exporting to utilize PPI ad testing. – Eric

    • brad
      February 21, 2013 at 12:50 pm · Reply

      We’re working a software that will make this very easy in the future. I’m hoping to have it released in a few months. I think PPI is a great metric, but you’re right; it can be hard to calculate at scale or within the AdWords interface. Even CPI (conversion per impression) would make a good firs step.

  4. stef_vliet
    April 26, 2013 at 2:37 am · Reply

    Great article Brad, and i’ve been trying to use PPI myself. Although it’s really easy (and fun) to calculate PPI, I’m not sure how to determine if one PPI improvement is statistically significant. How would one determine this exactly?

    • brad
      April 26, 2013 at 7:34 am · Reply

      Great question. You’d do it the same way you calculate it for CTR, CPA, etc. However, you’d used PPI instead of the other metric in the calculations. The one issue I’ve run into is that $0.01 vs $0.015 is so small (yet over 100,000 impressions is a decent number); sometimes I have enough impressions/clicks to know that it’s SS; but as those numbers are pretty small – some of the simple tools don’t actually say you have SS.

      So, you can either do a lot of math; or use some simple rules like:
      Do you have at least 1000 impressions per ad?
      Do you have at least 20 conversions per ad?
      If you extrapolate the profit, is it significantly different?

      That isn’t perfect math; but easier for some to deal with.

  5. drake123
    May 1, 2013 at 12:18 am · Reply

    Do you consider this a metric that should pass to websites and traffic to them as well? (Profit per Impression/Visit?)

    • brad
      May 1, 2013 at 7:43 am · Reply

      Not sure if I totally get the question.

      It’s a metric that you test post ad/landing page click – so its not just the ad being tested – its the ad and landing page in combination that’s being tested.

Leave a Reply