We all know of the importance of measurement and evaluation but in this time and resource-poor era are you focusing too much on the easily countable at the expense of the true picture? Panel writer Sarah Lay looks at the danger of McNamara’s fallacy in the evaluation of communications.
by PANEL WRITER Sarah Lay
There is a saying that I am fond on when looking at measurement and evaluation of digital and communications: ‘A number has no meaning other than that which you place upon it’. It’s a reminder that while qualitative data may well be easy to collect, it rarely tells the true or full picture. While the headline figures may be made attractive by time pressures, without that full story how well informed can your future efforts be?
The McNamara Fallacy
Relying too heavily on that which can be easily gathered and stacked up has become known as the quantitative fallacy, or the McNamara Fallacy.
Robert McNamara was the secretary of defence during the Vietnam War and developed an approach to measurement based on factory production lines. It relied on assumed predictable and repeatable behaviour, as well as advising to discount anything which wasn’t easily counted.
In a move that saw body count being used as a measure of success it was blind-sided by the messy chaos of human behaviour – a realisation he was reportedly haunted by in his later life.
“The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can’t be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can’t be measured easily really isn’t important. This is blindness. The fourth step is to say that what can’t be easily measured really doesn’t exist. This is suicide.” (Yankelovitch 1972)
High numbers don’t mean success
Those easy numbers are the ‘new website draws double the daily visits’ headline. But if you only count visits and look no further you are unlikely to understand the real issues and opportunities, the successes and failures of your current offer.
So – your web page is bringing in a good amount of visitors, perhaps you’ve even been able to show this has corresponded to a sought-after uptake of a service, or a reduction in contact through other channels. But have you looked into whether that page is meeting the user’s need? How sustainable is the change if your content fails to satisfy.
You need deeper stats here about bounce rates, time spent on page, completed actions, user comments. If people were following a search result were the keywords they entered relevant to your page – if they can’t find the answer does a larger number of them visiting mean anything (other than maybe a large number of frustrated people)?
Perhaps you persuaded someone to sign up to your mailing list but they left within a week, or they never open the mail you send to them – in which case, was it ever really success? Or the video you posted on Facebook got 5000 views, a handful of Likes and a couple of shares. But how many of those engagements resulted in an action?
Maybe your content does ok and achieves what you want it to most of the time, but by looking at the detail could you make an improvement that takes it from functionally meeting a user need to satisfying extra desires as well, from that passive engagement to action.
With shorter time and greater pressure to prove the value of your efforts it’s tempting to take the number which looks good and pin your value to it. This might get your short-term praise internally but may mean you’re missing the chance to really understand how you’re meeting a user’s need, and how you can do even better next time.
Sarah Lay is a digital content strategist, consultant and comms2point0 Lifetime Achievement UnAward Winner, with experience of delivering across the public and private sector. You can find out how to work with her, and read more from her, on her website www.sarahlay.com or find her on Twitter @sarahlay.
image via Thels Kofoed Hjorth
