Learning from surprises: the need and several obstacles

What looks as a big surprise is actually an opportunity to learn an important insight about our reality. Many times we ignore the opportunity and in other cases we learn the wrong lesson. We should learn how to learn the right lessons.

Bad surprises, but also good surprises, signal us that somewhere in our mind we have a flawed paradigm.  A paradigm is a small branch of cause-and-effect tree that we automatically treat as true, while in a case of a surprise it becomes clear that the tree contains a flaw.

The first task of learning from the surprise is to verbalize the gap between prior expectations and actual outcome.

I like to use the latest news about the US presidency election. The focus is not on any political aspect. My focus is on the failure of ALL the polls.  The gap is: “We were led to believe the prediction implied by polls”, but the actual outcome was the opposite – creating a big surprise that raises many practical questions for both the political arena and also for managing organizations.  The most obvious lesson is to stop believing in polls in general. Is this a valid lesson???

Such a lesson has huge practical ramifications. After all statistical models, like those at the core of the polls, are important tools for managerial information that lead to many decisions.

This is the first serious obstacle of learning from experience:

There is a big risk of learning the WRONG lesson

The first lesson in dealing with surprises is:

Do not jump into fast conclusions!

Instead come up with several alternative explanations before carrying a cause-and-effect analysis, leading eventually to the right lesson, which leads to practical consequences that are based on improved understanding of the cause-and-effect in our reality.

For instance, it is possible to claim a “rare occurrence” leading to the failure. After all no poll claimed the predicted result with 100% confidence. Even when the claim is 90% then there is 10% chance for a different result.

The hard fact is that we cannot prove the “rare occurrence” theory is wrong. However, in this particular case the failure of ALL the polls in the US is added to similar failures of polls in the UK (the Brexit) and in Israel. It is still possible that all of them are rare occurrences, but it is unlikely.

Possible alternative explanations:

1. It is IMPOSSIBLE to predict what people would do by simply asking them direct questions before the event. In other words, we don’t know how to predict the reaction of mass of people. Note, that if this is true then all market surveys are useless.
2. It is possible to have a credible poll, but the current method has a basic common flaw. This hypothesis is actually a category of several, more detailed, assumptions regarding the flaw.
   1. The way the statistical sample is built ignores, or under estimate, relevant parts of the society.
   2. Many people deliberately lie to polls.
   3. Many people change their mind in the last minute.
   4. Many people are influenced by an informal leader, so they have no strong opinion of their own.
   5. Asking people about their preference does not mean they will take an action (like going to vote or buy anything) accordingly.
3. Carrying polls creates a new situation where many people react to the poll in a way that changes the result.
4. Polls rely too much on mathematical formulas, without considering enough the impact of the communication with people answering the poll.
5. The polls are grossly misunderstood as they never say something concrete and people do not understand the meaning of “statistical error”.

Each one of the possible explanations should be analyzed using cause-and-effect logic, including looking for other effects that validate or invalidate the explanation. Eventually we get an explanation that is more likely than others.  Even here we should tell ourselves: “Never say I know”, but the analysis, in spite of its level of uncertainty, gives you much higher chance of enhancing your life than by telling yourself “I don’t know anything”.

Generally speaking identifying the flawed paradigm does not complete the analysis. One needs to come up with an updated paradigm and then analyze the practical ramifications of the updated paradigm.

I don’t have the knowledge to carry a full analysis of the failure of the polls by myself. This is a process that requires people who have been involved with the polls together with others who can spot and challenge assumptions.  Question is:  would such a serious process take place?

Two different groups should be interested in such an analysis:

1. Politicians, campaign managers and also C-level managers, notably the heads of Marketing and Sales.
2. The statisticians, both academic and professionals, who carry those polls or impact them through their work.

Here is the truly big obstacle to learning:

Do you really want to recognize your own flaws?

This obstacle is emotional; there is no practical rationale behind it.  But, it exists in a big way and it causes HUGE damage.  It impacts our repeated failures in our life and it never generates anything positive.  We like to protect our ego – but our ego is NOT protected by that at all.

Eventually, it is a matter of our own decision to look hard at the mirror.  Some would claim it is not possible to overcome such strong emotions.  It is a good excuse.  Somehow I have seen people taking this decision and going quite far with it – maybe not to every flaw they have, but overcoming some of them is already an achievement.   One of them, by the way, was Eli Goldratt, who clearly struggled, and eventually succeeded, to be able to compliment other people on their achievements.

If you like to know more about learning from ONE event – go on this blog to the main menu, enter Articles, Videos and More, and read the second article entitled:  “Learning from ONE Event – A white paper for expanding the BOK of TOC”