The Importance Of Critical Thinking When Reading Industry Posts

With the most recent AdWords shakeup, the removal of right column ads, there have been the expected reaction articles and posts written. Some were quick takes that were on the fly reactions and some have been written to declare the impact of these changes. Reading these pieces, and other pieces that are presented as if they are universally applicable, has gotten me thinking about a very important skill – critical thinking. When authoritative companies, sites and individuals write these kinds of pieces, they are often taken as the gospel. I don’t think anything is everyone’s gospel and wanted to offer some thoughts on reading, digesting and implementing findings or advice with a grain of salt.

Let’s start with the right column ads. The change was made by Google a mere month ago and already there have been quite a few posts written that claim to quantify the effects of the change. Some are better than others when it comes to qualifying their facts or reiterating that they are merely reporting on what they have seen SO FAR, but all are offering conclusions. I find this fascinating for an industry that is so data obsessed that anyone would be writing about what this change means in any kind of concrete way with such a limited data set. We often talk about having patience to wait for enough data in our clients’ accounts to make meaningful decisions – we need to do the same when it comes to what the long term impact of a change like this will end up being.

Four weeks is not long enough for major brands to have made big time strategy changes. Four weeks is not long enough to know how competitors will react over time either. When you are reading posts that draw conclusions, make sure you understand all of the underlying assumptions, methodologies and data points.

Why Averages Are A Sucky Metric

Reading about average spends, average conversion rates, average clickthrough rates, etc. it is easy to be lulled into thinking that average equals universal. It does not. In fact, averages can be a really dangerous figure to build your strategy around. Consider this example using a small sample:

Change in bid cost by percentage

  • Client 1: 0.5%
  • Client 2: 1%
  • Client 3: 1%
  • Client 4: 1.5%
  • Client 5: 2%
  • Client 6: 12%
  • Client 7: 15%
  • Client 8: 0%
  • Client 9: 2%
  • Client 10: 1%

If you took the average change across these 10 clients, you would present that change as averaging 3%. Great, except that assigning an average like this to a diverse group of figures such as this is deceiving. Multiply this effect out across thousands and thousands of accounts and the math problem is still there. An average is a terrible way to see how your account is stacking up. There is absolutely no room in just calculating a straight average across accounts for a metric for any other factors that are actually important in the real world. Looking at the numbers above, you can see that the average without the two outliers is closer to 1%. But the figures that would be quoted when looking across a sample like this would be 3%.

You Need To Know More

Personally, I don’t find articles that provide findings across a huge sample all that useful. From my experience in managing PPC accounts for quite a number of years, I can tell you that no two accounts perform exactly alike. Even those in the same industry can behave very differently in different geographies for example. When numbers are presented as monoliths, it can set up dangerous benchmarks that are not accurate for your account. It does not seem like many posts offer this disclaimer though. Rarely do they even offer a demographic profile or cross-section so that you can get a sense as to whether the data being discussed matches up with any accounts you are weighing it against.

I always want to know things like:

  • What is the average (cringe) spend of the accounts included?
  • What are the industries included in the data set?
  • Are the sites ecommerce, B2C, B2B, etc.? What is the percentage of these types?
  • What are the levels of competitiveness of the included industries? Are they similarly competitive or all over the map in competitiveness?
  • How expensive are the clicks in the included data set? What does the curve look like for the CPCs – bell or spiky?
  • Are these accounts national, regional, local or all of the above? What is the split and do the numbers differ when you break them out by geo target size?
  • How tightly are they managed? Are the professionally managed or DIY?
  • If discussing conversion rates what percentage use targeted landing pages?

These are all examples of critical thinking – not just taking what is put before you as truth because it was written, spoken or shared. What you want to be able to do is take information that is available and decide if it is relevant to the situation YOU are managing. There are few if any universal truths for PPC. Each situation, while it may follow a basic framework for setup of management, is unique. The client’s specific needs, goals and anxieties are unique. The market in which they compete is unique, be it industry or geography. If a study was based on account profiles that do not match your account, the advice or conclusions being offered might not be valid for you. Knowing how to filter information against these kinds of criteria is the essence of critical thinking. It is an incredibly valuable skill to have.

What are your first thoughts when you read these types of pieces? What types of additional information do you wish were presented in these types of posts? Do you have a particular process you use to evaluate the veracity or applicability of published posts for your accounts? As always, sound off in the comments or hit me up on Twitter (@NeptuneMoon).


  1. I once heard Brad Geddes say “Averages lie.” Some of the best advice I’ve ever heard.

    • Neptune Moon says:

      I have also heard him say the same thing on more than one occasion and found myself nodding enthusiastically. Part of why Brad is so fantastic!

      Thanks for sharing your thoughts.

  2. Great read and I agree, especially on knowing the type of PPC account (retail, B2B, local, etc.). And there are so many other variables as you point out, that it’s just hard to take it as universal.

    • Neptune Moon says:

      Thanks for commenting!

      Glad to know I am not the only one who has these thoughts and questions when I am reading posts.

  3. As we always say in PPCchat – “it depends”. Good article and I agree with most of it, But I do like to see these studies, too. Just as an insight to what others may be seeing. I don’t necessarily use them for strategy changes, but I do feel they have some value from more of a curiosity standpoint. But to your point, you can’t/shouldn’t make them the basis for your decisions. Use your own data, which you can slice and dice.

    • Neptune Moon says:

      I like seeing others’ data too for the same reasons. My other worry, that I have encountered all too often, is when non-pros read these posts and don’t know how to process the information. They take it as universal truths or benchmarks and that can be disastrous.

  4. At long last! Someone said it! Kudos!

Speak Your Mind