Rails, Scrum, CMMI and Survivor Bias
The biggest problem with tales about software success is that they only show the survivors. This problem is independent of the domain, be it Rails, Scrum or CMMI success stories. Survivor bias is when you only look at the survivors of a process and then attribute the success of the survivors to some attributes like cleverness or strongness.
Survivor bias is most easily shown with financial funds. Say you want to examine the success of some financial funds. You look at funds which are investing in high tech stocks.
When looking at nine funds with the growth rates of 7.3, 9.2, 12.1, 5.4, 6.7, 7.2, 10.1, 8.5, 8.8 we get an average rate of 8.4 percent.
This finding would suggest that funds which invest in high tech stocks have an average growth rate of 8.4. But assume there was one fund which defaulted. People looking at average growth usually only look backwards, which means they only select surviving fonds. When we go back in time and look toward the future, we see ten funds we want to measure. After some years, nine have survived and one has defaulted. So from a point back in time we selected ten funds, not nine. And with the tenth fund, with a growth rate of -100%, included in the average we get an average growth rate of -2.5 percent for the ten funds compared to 8.4 percent for the nine surving funds. That’s survivor bias.
The same can be said about other processes, most notably progessing of CEOs and companies, especially start ups. When looking at successfull startups people are mislead to attribute the success of a startup to some genius founder, good products or other attributes when in fact, an equally good explaination is that the startups had only luck (for that we need to get back in time and look into the future with all the startups, not look back from the point of successful startups). The same goes for CEOs: most could be successfull by luck, the unlucky ones just get kicked out faster and don’t show up as often as lucky ones.
Coming back to software development. We hear lots of success stories from companies and developers about tools, languages, frameworks, plattforms and processes. Those people praise Rails, Scrum or CMMI. But looking at funds and transfering our knowledge to those praise stories, we could assume that those involved were just lucky.
Companies which failed with Rails, or Scrum, or CMMI and the countless other solutions for software development seldom talk about their experiences.
- Companies which fail with Scrum just drop it.
- Companies who try Rails in a pilot just don’t use it.
- Companies who are not able to get their CMMI certification just don’t start it.
And some of the companies even didn’t survive to tell their tale. Most people publish success stories on their blogs, in magazines or on websites because people want to read success stories and success stories make the authors look good. Failures make them look bad. This leads to survivor and publish bias.
The same is true for business advice:
Doesn’t most business advice suffer from this fallacy? You read about successes but what about the businesses that “never made it home?”
Next time you hear a success story, think about all the people who don’t or couldn’t tell tales about their failures. The success might just be plain luck and survivor bias.
You can leave a Reply here. Of course, you should follow me on twitter here.
Excellent analyses and well applicable to many subjects (eg, Offshoring).
Especially in a business enviornment failure is never (publicly) acknowledged, so the statistical sample is skewed as you point out.
Jordan