Does Quality Score Transparency Just Show More AdWords Data Problems?
Google recently changed how they displayed the Quality Score tips inside of AdWords. The biggest change is that some of the factors will now be relative to your competition. While this is a huge, and very welcome change, it also opens Pandora’s box to show where quality score might not make much sense.
How Much Does Landing Page Matter?
First off, landing page speed is no longer being displayed. That’s not really a huge issue as I’ve never seen a problem with landing page speed. They instead rolled the landing page issues into one single ‘landing page relevance’ score.
For years, if your landing page was dinged by quality score, you were in trouble. Rarely would you ever see a quality score above 3 or 4 if you had landing page issues. That no longer seems to be the case.
Now, when I see a 10; this is what I was expecting:
But this is what I was seeing across several accounts as 10s as well:
I was floored when I saw this keyword. A below average landing page experience, and yet the quality score was a 10. It didn’t take me long to find dozens of examples where this was occurring.
The problem with relative data is that Google doesn’t tell you what fits into above or below average. For example, if average is 1, is below average 0.99 or 0.9 or 0.8. If average was a .8 to 1.2 range; then below average is meaningful. If below average is 0.99; then below average might not mean that much.
It is Mostly CTR
For a long time, landing page was more of a negative than a positive to your quality score; and it was all about CTRs. That seems to be the case. In this example, everything is average or above average; yet the QS is still a 4:
So, in example 1, the landing page was below average, but the CTR was good enough to get a 10. In this case, having an expected CTR above average, and everything else as average meant the QS was a 4. That seems quite counterintuitive; especially when you see this keyword which is also a 4:
There’s no way that both of these keywords should be a 4. The first one is all above average or average. This one is below average. This should probably be a 4; but not the first one.
‘Relevance’ Still Matters
While relevance is technically another set of CTRs, its usually best to think of this as semantics. And they matter:
The majority of quality scores I saw at a 2 had issues with relevance, not CTR.
‘Average Ads’ Can be 3s to 10s
Google has made enough claims over the years about 7s being good and 6s needing a bit of help; but what is average?
I’m seeing words from Quality Scores 3 to 8 where everything is ‘average’.
I’m seeing quality score 7 words that are all average, or 1-2 items is ‘above average’.
I’m seeing quality score 9 words that that have less above average items than quality score 5 & 6 words:
You Can’t Diagnose Paused Words
If a word is paused; the metrics will all be below average. This does lead credence to the theory (that I subscribe to) that a paused word can’t hurt you. Google isn’t collecting metrics; therefore it’s below average as there’s no data coming in. A better error message here would be a good idea instead of just ‘below average’.
So, What Can You Takeaway?
A good landing page is necessary for conversions. A bad landing page (in Google’s eyes – not the searchers) could have a negative quality score affect; or it could not.
An ad with ‘below average’ expected CTR can have a quality score of 1 to 6. I didn’t see any quality score 7 or higher words with below average expected CTR.
Ad relevancy matters. You can have a 10 with a high relevance, but average everything else. I didn’t see any higher quality score keywords with below average relevancy.
The ‘average’ benchmarks seem to be different for each of the displayed data sets. The fact that landing page can be below average and get a 10; yet relevancy and expected CTR can be average and be anywhere from a 4 to a 10 is either an error in the quality score algo; or the ‘average’ range is quite large.
In the end; while these numbers are relative; I think we need a better scale than average, above average, below average. A 1-5 range (if you want to make it semantic with bad, poor, average, good, excellent – that will also work, Google) would be much more insightful.
The quality scores I’m seeing don’t make sense in many cases (and I only spent 10 minutes looking for these examples, they aren’t the strange ones – they are the norm); and I think it’s a range issue.
So, kudos to Google for showing some relative data; however, my hope is that Google goes much further with the ranges. The new quality score transparency is not that useful and will raise more questions that it answers.
this is how Google explains away what you’re seeing:
https://support.google.com/adwords/bin/answer.py?hl=en&answer=1659694
“It’s possible for a keyword to have a high Quality Score and low landing page experience (or vice versa) because AdWords looks at a number of different quality factors when determining Quality Score. Even if your overall Quality Score is high, looking at the individual factors can help you identify potential areas for improvement.”
I think we need some kind of number scale for these factors. At least give me some indication of how bad or how good I’m doing.
“It’s possible for a keyword to have a high Quality Score and low landing page experience (or vice versa) because AdWords looks at a number of different quality factors when determining Quality Score.” Yes – so if you can have a below avg LP but a really high QS, then it follows that LP does not factor much into QS (but as always with the caveat that if your LP is *really* bad, it will have a huge effect on QS).
A very interesting post, thankyou. I agree that Google’s classifications of what “Average” is, is vague to the point of uselessness. Giving them numbers would be a much better idea, and I would suggest a 1-10 which most people accept as a system of marking. After all, in school you know you did well with a “9”.
The bigger problem is that Google is trying to do something with a machine that really lies in the emotional realm: choice. What is a good landing page, and what is not? Much of it is down to aesthetics – and that cultural as anyone marketing outside the US will have discovered! Indeed, even within the US there are substantial geographical differences. That is, after all, where split testing comes in – and our own lack of appreciation of who we are leads to us choosing what would be rank outsiders!
That is something that a computer is unlikely to have much experience of. Sure, they are assembled in China, the semiconductors made in Japan or Taiwan … but that does not mean they speak Chinese. They will interact in whatever language you program them in.
Actually, recently Google changed their style in their Keywords tool, where competition is described in three terms rather than as a block graph. The latter was far more useful as a metric.
Given that Google make these decisions on data that they have for your landing page, surely it would be possible to have a neat little flag. The red, yellow and blue showing not “average” but a proportional result.
Google’s idea of what “Average” is or means is entirely personal to them. Being Google, you are never likely to be told precisely what that stands for.
I was told by many a Google Account manager that your quality score is effected by competitors in the auction. So when Google use the term “average” (below or above), What is the benchmark? Is it below average in terms of my account or across all accounts bidding on that word? I have accounts in verticals where a quality score over a 3 is impossible to get. When I have questioned this with Google I was told it was an over abused Keyword (What ever that means). This made matters worse for me as I felt my keywords, landing pages & ads where super tightly targeted. Anyway, take from this what you will… Just thought I’d share.
Thanks for breaking this down Brad. Hopefully, Google will continue to respond and improve the transparency over QS. I’ve certainly had some real head scratchers in the past; I’m sure there will be more to follow! Your insight and guidance is a great help though.
BTW, I just received the second edition of your book today so I really have no excuse for not knowing! 😉
I think the new quality score transparency reinforces the notion that landing page has little influence on QS, as has always been the case. If you can get a QS of 10 with a landing page experience that is rated as ‘below average,’ then the landing page must not be a big part of the QS algo.
I think Google keeps it simple when deciding on landing page quality. If data is collected on a landing page or subsequent page, is there a privacy policy? Is the page broken in some way, or does it load very very slowly? Or, are there any indications that the page is an old-style affiliate page whose main purpose is to direct traffic somewhere else? It’s quite clear that not all landing page experience ratings of ‘below average’ are equal; we know that LPs that are deemed truly bad will get you slapped.
The new transparency can be helpful where it shows CTR and/or relevance as average or above average, but where the QS is low. If CTR or relevance is average or maybe above average, you are doing the best you can with this keyword. I think that QS is relative to competing advertisers, but it also takes into account the performance of all ads for that keyword over time. Some keywords will never show a high QS for any advertisers because of the nature of the user intent (or lack of clear user intent) behind the query. A kw with above average CTR or relevance but with low QS reinforces this notion IMO.
CTR is still king; using the keyword in the ad seems to give you a little QS bump; landing pages – unless Google deems them seriously poor – have little to do with QS; and, most importantly, QS is much, much less important than optimizing for performance. There is oftentimes an inverse relationship between QS (read: CTR) and conversion rate. It’s typically pretty easy to write an ad that will get a higher CTR – but those ads will often have lower conversion rates. The trick is to write the ad that gets you the most conversions per thousand impressions, at the lowest cost/conversion. None of these things have changed.
The reason I was so surprised about the landing page is that when its bad – its a huge factor.
I’ve seen large spenders (multi millions per month) have their account suspended due to landing page quality scores. I’ve seen other large spenders hit with domain level penalties.
It was rarely a positive; but it was a huge negative.
Now, I think that’s still true. I think that a ‘minor’ below average landing page isn’t really ‘bad’ in the old world – its just below average in the new one. However, a ‘major landing page penalty’ isn’t displayed anymore; its just lumped into ‘below average’.
I don’t think how QS is calculated has changed that much; I think its the way the data is presented has changed a lot.
yes, agreed!
I have a client where the exact-match branded keyword – just one word – has had an average CTR of 20-30% over many years, and current QS for this branded keyword is a 7. Hovering over the bubble reveals that everything is ‘above average.’ Doesn’t make a lot of sense, as many things related to how Google displays quality scores make little sense.