A while back, I read an absolutely fantastic post by the authors of Freakanomics (Steven Levitt and Stephen Dubner). The post was titled ‘The Dangers of N=1’ For you non-stats folks, ‘N’ (or ‘n’) is sample size – the number of samples you collect in your experiment. If you have a ton of samples, and draw a conclusion using the correct statistical method of analysis (another potential problem in drawing conclusions), you can be reasonably sure that – statistically speaking- the conclusion you draw is true or repeatable. However – and this is the thesis of the post – what happens when we as humans draw inferences from a small data sample – like 1? In the post:
… Consider the numbers: At 35 years old, I figure I have conservatively showered well over 12,500 times in my life, and I have only found one spider in my towel. And yet, for the last three weeks or so, I have been checking my towel every time I shower.
It got me thinking: How often do we allow our behavior to be influenced by single data points. Are there any positive examples?
It’s happened to me… it’s probably happened to you. Something happens, but it’s rare. The outcome is startlingly enough to make you think that it could happen often. So you prepare for it – but you’re basing your actions on one outcome.
Go read the post. It’s solid, and well worth the 5 minutes of your time.
Like a good social media doobie, I shared this post on Facebook and G+ and what not. A friend of mine (a brilliant and published scientist) read it as well. Along with a couple of degrees, he’s got years and years of experience in the lab running (statistically valid) experiments. Moreover, he also has the experience of presenting these findings to executives, peers, and other decision makers. Best of all, he’s a human, and you can actually have a conversation with him. This is important, and you’ll see why in a moment.
He read the post, and shared a story with me. Below is a paraphrased excerpt:
I have come across in different industrial and academic organizations that based their research by wasting many dollars surrounding the realization that their N=1 data was a truly astounding breakthrough, and that many dollars should be wasted spent on researching the “next steps”. Instead of repeating the experiment to see if it’s an anomaly, or to prove that something was done incredibly wrong, some scientists like to jump to conclusions to prove their initial hypothesis was indeed correct. The “N=1 scam” in science, in my opinion, is usually used to prove oneself right and make themselves feel and look smarter.
These same folk concoct crazy experiments that do not systematically work toward in either proving or disproving the initial data and just move on, again wasting time and money. Often these experiments come without proper controls (if they even have controls) and are completed merely to prove themselves correct. Some will go so far as to keep doing this. The next round of crappy data will not support their initial claim but they’ll use statistics (sometimes the wrong method) to bend the data to support their hypothesis.
It happens all the time in everyday life. Somebody hears of some supposed evidence (n=1) that vaccines are bad or something and they automatically discount all of the data that is staring them in the face. Forget actually educating oneself and looking into the data that’s out there. They’ve always thought that the medical industry is out there to get them and – boom – “they’re right.”
The fact that this occurs in science – the very science that works on curing cancer, solving world hunger, creating resistance to H1N1… startles me. Believe me – I get the desire to want to ‘own’ a field, to want to prove one’s worth, to show that one isn’t an intellectual dud. But to do so at the expense of (potentially) millions of research dollars? Or the next round of funding (if one is in a startup)?
There’s a certain cross-over point from the wet-eared, undergrad (or grad) researcher, who wants to change the world, to the near-maniacal tyrant, with hubris that fills the lab and then some, that makes calls about research and ‘fact’ that can’t be substantiated. When does this cross-over happen for some? What dictates it? What are the contributing factors? The signs?
Moreover – if you are in the presence of this, and know (or ‘feel’) that anything contrary could be possible… how do you raise the issue? How do you do so without coming across as a challenging jerk? Or doing so without fear of retribution?
This isn’t a technology problem. This is an issue that deals with the human interactions and organizational behaviors of those inside technologically forward roles. I’m curious to hear and read your thoughts.
image source: http://www.flickr.com/photos/talllguy/2432658473/
If you like what you’re reading here, consider subscribing to the RSS feed or signing up via e-mail (upper right corner) to be alerted when more new posts are added.