Survivorship Bias, and learning from the fallen
- Vishal Barfiwala
- Mar 10, 2022
- 9 min read
The story of Abraham Wald
The scene is set in 1943 - at the peak of World War II in New York. America had set up a classified program in Manhattan, at 401 West 118th Street in a building called Morningside Heights. Unlike what we would typically associate with a 'classified program' though, this building was filled with mathematicians and scientists in groups such as the Applied Mathematics Group (AMG) - which defined the trajectories the fighter planes had to take, to another group which defined protocols for strategic bombing.
One such group housed in that building was the Statistical Research Group (SRG), which was considered as the most elite group. This is where our protagonist Abraham Wald was employed. Wald was born in 1902 in Klausenberg in Romania, and was a mathematician who had immigrated to the US and became professor of statistics at Columbia, soon after which he joined the SRG.
One of the most famous problems faced by the SRG is that of optimally armoring the fighter planes. Once the fighter planes went to Europe during the war, only 50% came back because they were shot down by enemy fighters - and so the problem was to figure out how and where to add armor to the planes to improve survival rates. Too much armor would make them heavy, less maneuverable and fuel inefficient, and too little / in the wrong places would not save them from enemy hits.
The military which came to Wald came with data - the density and a plot of the distribution of the bullet holes from the planes which came back from engagements over Europe. An image of the distribution is shown below.
Location of the hits in the aircraft. McGeddon, CC BY-SA 4.0, via Wikimedia Commons
If you were to solve this, what would your answer be? Where would you add the armor?
The military team had it almost figured out - of course the armor needs to be added in the areas with the highest bullet hole density. The question they wanted Wald to solve was - just how to optimize the armor's protection vs. weight in those areas.
Wald gave his answer, just not what they expected - He said, "the armor doesn't go where the bullet holes are, it goes where they aren't. On the engines". (This is indicated in the image below.)
The rationale is simple - the data on bullet hole distribution was based on the planes which returned - the survivors. If it is fair to assume that the distribution of bullet holes were to be uniform, the question Wald had in mind was - where are the missing holes on the engine parts? And the answer was clear - they were in the missing planes which never returned - the fallen, the ones which didn't survive. Which is exactly what they were trying to prevent from happening. Following is an excerpt of his analysis:
What you should do is reinforce the area around the motors and the cockpit. You should remember that the worst-hit planes never come back. All the data we have come from planes that make it to the bases. You don’t see that the spots with no damage are the worst places to be hit because these planes never come back.
These recommendations were put into effect, and have saved a large number of lives, even in wars after WWII - in the Vietnam war and in Korea.
Survivorship Bias - What it is
Survivorship Bias is the tendency to act based on the information from the survivors, while ignoring that of the fallen (or failures). It distorts our world view, and makes us ignore base rates and creates confusion between co-occurrence (coincidence), correlation and causality. And is one of our many cognitive biases.
All around us, we see stories about winners. People who made it big - in the movies, in politics, in business and startups, in life in general. No one talks about the hundreds of struggling actors who didn't get a role for every successful actor or the hundreds of startups which went bust or the authors whose books never got sold. The losers just don't show up.
Here's how Taleb described Survivorship Bias in his book - Fooled by Randomness:
In a nutshell, the survivorship bias implies that the highest performing realization will be the most visible. Why? Because the losers do not show up… The mistake of ignoring the survivorship bias is chronic, even (or perhaps especially) among professionals. How? Because we are trained to take advantage of the information that is lying in front of our eyes, ignoring the information that we do not see.
If you like the content, do subscribe and get notified of new articles directly in your inbox.
Examples of survivorship Bias
The college dropout conundrum - should I, should I not?
What's common among Bill Gates (Microsoft), Mark Zuckerberg (Facebook), Steve Jobs (Apple), Michael Dell (Dell), Ritesh Agarwal (Oyo), Jack Dorsey (Twitter), Travis Kalanick (Uber)?
They all dropped out of college to pursue their startup idea, and build successful (or well funded) companies. And there is a lot of media coverage on the fact that they are college dropouts.
We look at this, and I'm sure we know someone (maybe we ourselves?) who has thought about dropping out of college to pursue their passion / idea as an edge which gives them a head start in achieving their dream. The questions which should actually be asked, however, are - what % of successful companies / startups are headed by college dropouts? And What % of college dropouts are successful as compared to college graduates?
Fortunately, the answer to the first question has been dug out. In a study of 11,745 successful individuals from across the U.S., the vast majority had attended college -- many of them elite schools.
The statistics are plotted below:
Now dropping out of college doesn't seem that lucrative, right?
Fund returns are subject to market risks, and survivorship Bias
Most analyses of mutual funds or hedge funds are based on analyzing returns of funds which have survived for the period of the analyses. Funds which have closed, usually due to poor performance, during the period of the analyses are generally ignored, due to lack of information. This leads to a positive bias in the returns of the survivors being analyzed and generalized for the category.
There have been studies which have tried to quantify the overstated returns of funds, after factoring in the failed funds. One of the comprehensive ones in 2011 in the Review of Finance covering over 5,000 funds showed that the excess return of ~2,650 survivors is 20% more than what it is with even the fallen firms considered.
The next time someone throws average fund performances to you, don't take them at face value, and ask if this is based on the survivors alone, or if it includes the fallen.
Social media amplifies Survivorship Bias
The beauty, brain and brawn on social media is as biased as it gets, with survivorship
All of us follow some or the other actor / influencer / friend who posts insanely beautiful photos on social media (read Instagram) - the ones that you can never get how hard you try. The ones with beautiful scenery, or the best interior design, or the perfect makeup and elegantly dressed. Well that is because the ones which are posted are survivors among hundreds of failed attempts. The amount of time and money Influencers spend to create their posts is astonishing. Further, these posts are survivors among the best moments of their lives which people like to share with others on social media. A double survivorship bias if you will.
Similarly, you see people posting about phenomenal returns they make on stocks / crypto / F&O and it makes you wonder if you are really doing something wrong. Though, if you think about it - only the ones who make good returns would post it on Twitter. In fact, even these people would only post when they make good returns (and stay mum when they make bad returns). Once again a double whammy.
So the next time you get depressed comparing your mundane life, boring job, low returns etc. with the exciting lives of the people you follow on social media - just come back and read this article, trust me, you'll feel better. (And subscribe too!)
Most studies analyzing what are common features among good companies are again prey to survivorship bias. For all you know, bad companies could be doing the same things and still being bad as well.
If you like the content, do subscribe and get notified of new articles directly in your inbox.
How to survive the survivorship bias
And we could go on with such examples. However, now that we know what survivorship bias is, and how it could impact us in different areas, let's look at what we can do to overcome this bias.
Look for the fallen and learn from failure
Just as Wald, in our story, looked for where the bullet holes were missing, we should try and look for what is missing. Look for all those who started off on the journey, but didn't reach the end. Try and ask what happened to the failures. And then take a decision based on the complete picture.
If someone shares tools, habits, methodologies etc. just by studying the most successful companies, businessmen, startups, celebrities et al, please take them with a bucket of salt. There could be 10x more people following the same tools and habits, but are ignored due to the survivorship bias. Take, for instance, the 2001 bestseller "Good to Great' by Jim Collins (highly recommended by one of my ex-bosses). Collins analyzed the 11 best performing companies (from ~1,500) whose stock returns beat the market over a 40 year period, for identifying characteristics which made them successful. As you would realize, this exercise was highly influenced by survivorship bias, and as luck would have it, 6 of those 11 'great' companies underperformed the market from 2001 to 2012.
Consider base rates
What percentage of investors are on Twitter? What percentage among those would you be following? Is this a representative sample of the investor population? Answer these questions first, and then make your inference on the stupendous returns you may be observing on Twitter.
Base rate refers to a percentage of the population. For example, 50% of humans are male, or 80% of MBAs are engineers etc. Once the survivorship bias comes into play, we often focus on a sample (of survivors) who are not representative of the base rate. And hence err in our inferences. (There is also a bias called the base rate neglect, which I think is a fairly close cousin of the survivorship bias.)
Try to imagine alternate histories
Luck plays a role in success. In some cases it plays a small role, in some cases it pretty-much determines the outcome. For instance, in the game of chess success is almost completely based on skill, in cricket or baseball there is a good mix of skill and luck which determine the outcome, while in roulette or slots there skill plays no role and success is completely luck based. However, unlike these examples, it is really difficult to quantify or even broadly gauge the impact of luck versus skill. How then, do you know what to learn from the survivors - who have succeeded? What if they did everything wrong, and still succeeded?
One trick is to have a thought experiment and imagine alternate histories - what if some variables in life had panned out slightly differently, especially the ones which are influenced by luck. Would the person / organization still have succeeded in a majority of these alternate histories? If yes, only then should we learn from these survivors. A note of caution though, It is not easy to do this because you have reality staring at you, and you have to ignore it, to imagine alternate histories. However, if possible, it definitely helps.
How does this translate to investing?
Ignore most people on Twitter, Facebook, Reddit etc., who claim to have made superior returns - drown out the noise! If you really must follow someone’s advice - look at his / her long term track record (over several years or even decades). Nature and time generate a number of actual histories (as against our thought experiment of alternate histories for a shorter time frame) where the randomness of luck largely evens out over time, and what you usually see is hopefully the impact of skill. Which is why I think learning from Buffett and Munger is generally helpful.
While there is no harm in learning from people who succeed over long periods of time, it is harmful to set expectations based on looking at the survivors in a field such as investing where luck, undoubtedly, plays a huge role in success. Such people are tail events, whose success is difficult to replicate. Factor in base-rates to set your expectations. While you can hope, pray, aspire for high returns, you should expect and be prepared for - near average returns, unless of course, you have demonstrated superior skill over long periods of time with your strategy (and assuming you continue with that strategy over long periods of time, and it works).
Finally, when analyzing performance of an asset class / theme / set of funds etc., do remember the positive bias produced by the survivorship bias. Always ask if this performance being shown includes that of the failed and closed funds / schemes. Do not accept numbers thrown at you, challenge them.
Hope you liked this article on survivorship bias, and are able to take away a think or two from this. If you like the content, and think it adds value, please do share it with your network. And do subscribe to Investing in Rationality to get these articles hot from the oven.
If you like the content, do subscribe and get notified of new articles directly in your inbox.
Comments