What looks as a big surprise is actually an opportunity to learn an important insight about our reality. Many times we ignore the opportunity and in other cases we learn the wrong lesson. We should learn how to learn the right lessons.
Bad surprises, but also good surprises, signal us that somewhere in our mind we have a flawed paradigm. A paradigm is a small branch of cause-and-effect tree that we automatically treat as true, while in a case of a surprise it becomes clear that the tree contains a flaw.
The first task of learning from the surprise is to verbalize the gap between prior expectations and actual outcome.
I like to use the latest news about the US presidency election. The focus is not on any political aspect. My focus is on the failure of ALL the polls. The gap is: “We were led to believe the prediction implied by polls”, but the actual outcome was the opposite – creating a big surprise that raises many practical questions for both the political arena and also for managing organizations. The most obvious lesson is to stop believing in polls in general. Is this a valid lesson???
Such a lesson has huge practical ramifications. After all statistical models, like those at the core of the polls, are important tools for managerial information that lead to many decisions.
This is the first serious obstacle of learning from experience:
There is a big risk of learning the WRONG lesson
The first lesson in dealing with surprises is:
Do not jump into fast conclusions!
Instead come up with several alternative explanations before carrying a cause-and-effect analysis, leading eventually to the right lesson, which leads to practical consequences that are based on improved understanding of the cause-and-effect in our reality.
For instance, it is possible to claim a “rare occurrence” leading to the failure. After all no poll claimed the predicted result with 100% confidence. Even when the claim is 90% then there is 10% chance for a different result.
The hard fact is that we cannot prove the “rare occurrence” theory is wrong. However, in this particular case the failure of ALL the polls in the US is added to similar failures of polls in the UK (the Brexit) and in Israel. It is still possible that all of them are rare occurrences, but it is unlikely.
Possible alternative explanations:
- It is IMPOSSIBLE to predict what people would do by simply asking them direct questions before the event. In other words, we don’t know how to predict the reaction of mass of people. Note, that if this is true then all market surveys are useless.
- It is possible to have a credible poll, but the current method has a basic common flaw. This hypothesis is actually a category of several, more detailed, assumptions regarding the flaw.
- The way the statistical sample is built ignores, or under estimate, relevant parts of the society.
- Many people deliberately lie to polls.
- Many people change their mind in the last minute.
- Many people are influenced by an informal leader, so they have no strong opinion of their own.
- Asking people about their preference does not mean they will take an action (like going to vote or buy anything) accordingly.
- Carrying polls creates a new situation where many people react to the poll in a way that changes the result.
- Polls rely too much on mathematical formulas, without considering enough the impact of the communication with people answering the poll.
- The polls are grossly misunderstood as they never say something concrete and people do not understand the meaning of “statistical error”.
Each one of the possible explanations should be analyzed using cause-and-effect logic, including looking for other effects that validate or invalidate the explanation. Eventually we get an explanation that is more likely than others. Even here we should tell ourselves: “Never say I know”, but the analysis, in spite of its level of uncertainty, gives you much higher chance of enhancing your life than by telling yourself “I don’t know anything”.
Generally speaking identifying the flawed paradigm does not complete the analysis. One needs to come up with an updated paradigm and then analyze the practical ramifications of the updated paradigm.
I don’t have the knowledge to carry a full analysis of the failure of the polls by myself. This is a process that requires people who have been involved with the polls together with others who can spot and challenge assumptions. Question is: would such a serious process take place?
Two different groups should be interested in such an analysis:
- Politicians, campaign managers and also C-level managers, notably the heads of Marketing and Sales.
- The statisticians, both academic and professionals, who carry those polls or impact them through their work.
Here is the truly big obstacle to learning:
Do you really want to recognize your own flaws?
This obstacle is emotional; there is no practical rationale behind it. But, it exists in a big way and it causes HUGE damage. It impacts our repeated failures in our life and it never generates anything positive. We like to protect our ego – but our ego is NOT protected by that at all.
Eventually, it is a matter of our own decision to look hard at the mirror. Some would claim it is not possible to overcome such strong emotions. It is a good excuse. Somehow I have seen people taking this decision and going quite far with it – maybe not to every flaw they have, but overcoming some of them is already an achievement. One of them, by the way, was Eli Goldratt, who clearly struggled, and eventually succeeded, to be able to compliment other people on their achievements.
If you like to know more about learning from ONE event – go on this blog to the main menu, enter Articles, Videos and More, and read the second article entitled: “Learning from ONE Event – A white paper for expanding the BOK of TOC”
6 thoughts on “Learning from surprises: the need and several obstacles”
Interesting thoughts Eli. You pick on an important surprise of the polls. These could be wrong because they missed the enthusiasm of particular voters. Something that requires human judgement not just answers from the respondents. It will be interesting to see what the professionals say as to why they got it wrong.
Another surprise is what was the true appeal of Mr Trump despite his many flaws. Here again the conventional wisdom that he appealed to the worst in people does not ring true. This question is important because there was a real need that he captured that was being ignored by the mainstream politicians. This is an important learning needed as well. Any thoughts on that question?
Very nice! Regards Ray
Sent from my iPhone
I liked it too, Eli.
There seems to me a pattern that suggests Democrats poll better than Republicans in US presidential elections. For example, Al Gore significantly out-polled then not-President Bush. There was a similar surprise on election day and during the recounts in Florida in the days afterwards.
Likewise, Jimmy Carter was out-polling then not-President Reagan, who eventually won in what was termed a landslide.
Of course, sometimes Democrat favoring polls could cause Republicans to give up and not waste their time voting, possibly explaining other elections where the polls seemed to “get it right.” This suggests a motive for purposely slanting polls. I’d be surprised if that turns out to be the cause, however, if we ever discover it.
More likely, there is some difference between the two groups that nobody has thought of, for example, Republicans might be too busy to answer their phones during the evening hours due to being out to dinner or because they work late, making them under represented in the polls. Or, perhaps Democrats answer the phone and say what they expect to do but fail to vote as often as republicans do on election day.
Like any system with feedback loops, outcomes can be downright baffling, especially if one focuses on a local silo and tries to draw conclusions from data collected there. What a great lesson for us in business, government and families!
From the Wall Street Journal on Wednesday, “Not all models failed in predicting turnout and the strength of Mr. Trump’s vote. Both the USC/LA Times Daybreak poll and the IBD/TIPP presidential tracking poll predicted more favorable results for the president-elect throughout the campaign than other surveys did….Other political models at universities pointed to Republican victory or at least a tight race, Vox noted Wednesday.”
Maybe my main point was not fully understood: those who were SURPRISED by the actual result of the elections should do their best to find their flawed paradigm. The fact that few polls were closer to the result, but not many have listened to them, does not change the picture where there has been a major surprise!
And ask yourself another question: those whose analysis has been proven right – should we all listen to them next time? Are we all certain they know how to do it right, not just in one case, in every case?
Very pertinent point for all the decision makers – be it in public life or in business. Decisions are made based on past experiences/successes, risk analysis (statistical or paradigm-wise). A big opportunity for research to come out with better answers for future events.