The importance of Big Data

By Amir and Eli Schragenheim

Is Big Data important? Can every organization draw considerable value from it?

Amir and I assumed that the ultimate answer of most people in management would be: Yes, there is a big potential, but there is also a problem of drowning in the ocean of data (Goldratt in The Haystack Syndrome).

Well, as it seems too many people think that there is not much value to find in Big Data. So, maybe we, who think there is a very substantial potential value, need to back up this assertion.

Big Data in its narrow form is the ability of every organization to store huge quantities of data relatively cheap by the use of the cloud software tools for extracting specific data from various databases and formats, and organize them in a way that allows the human manager to focus on what is truly relevant.

A much wider approach to Big Data includes the huge amounts of data from external resources that allow free access through the Internet. Google, Facebook and LinkedIn provide the tools to do it and there are also public databases that allow searching and using their data for a certain cost.

It seems obvious that some organizations, certainly the bigger ones, are drawing a lot of value from Big Data, like the three big data manipulators mentioned above. Those giant organizations offer focused ways to advertise to well-defined audience.  Having the means to approach very specific market segments can be used to gain knowledge on the preferences of their customers.

The business sector of e-commerce, especially digital stores, is using their own huge data, taken from everyone who enters the website and records every move the user does, to draw conclusions on what the customer is interested in. The analysis of this accumulation of data opens a way not only for offering more to that customer with good chance for selling, but also winning that customer for future deals.  Beyond guessing the specific taste of every single customer, the generic understanding of groups of customers, like the role of price in their choices, can be established.

Physical retail stores use much less efforts to capture data that would reflect the clients’ preferences, beyond the trivial analysis of actual sales. Without direct access to client information, and even worse, without knowing what data could help them to gain more sales, they are helpless.  The retail stores lose a lot from their incompetence to collect the data they need to become more effective.

So, companies that have easy access to pretty straight-forward relevant data find answers to critical questions and gain a lot of value. Other organizations don’t.

When a new technology, like the ability to store and analyze huge amounts of data, presents itself to the market it raises two seemingly similar, but actually different, questions.

1. Given the existence of the technology can we utilize it to bring benefits?
2. Given our current obstacles – does the new technology lead us to overcome them? If so, what are the benefits going to be?

Many organizations don’t immediately see the benefits of a major new technology, meaning their answer to the first question is NO.

However, we believe some more efforts should be given to analyze what might overcome obstacles. Currently the organization accepts them as hard facts of reality, but the new technology is able to vastly reduce the limitation imposed by the obstacle. Then, new opportunities could be identified.

Goldratt 2^nd question, of the Six Questions for assessing the value of a new technology, states:

What current limitation or barrier does the new technology eliminate or vastly reduce?

The obvious limitation of storage is not the relevant answer to the above question, because the value of storing huge amount of data is not clear and could easily lead to waste of efforts. Also reducing the slow and cumbersome speed of collecting huge data and organize it in a friendly visible way does not always add value.

But, we always have the wish to have more relevant information on the critical issues the organization is dealing with. We never have perfect information when a decision has to be taken.  So, decision making is always under high uncertainty, due to variation, plus unknown facts.  While this basic life situation would continue in the future, the unknowns could be significantly reduced if the right relevant information is collected and given to the decision makers.

Thus, Amir and I suggested the following limitation/barrier that the new IT technology reduces:

Not being able to get reliable answers to questions that require data, which was before either unavailable or not accessible

For instance, what are the features that many customers miss in our current products?

It is possible to ask the customers such questions, and even store all answers, but many of them simply refuse to answer and maybe they do not know what they miss, but when they would see it, they will know. Can we answer the question if we analyze data on what caused certain products, from different sources, suddenly becoming highly popular?

Failing to answer critical question is a key limitation for every company, and a search for the truly relevant data should, many times, yield new information that, together with an effective analysis, should yield substantial value.

To clarify the sensitive connection between data and information let’s bring the definition Goldratt gave to ‘information’ in his book ‘The Haystack Syndrome’ from 1990:

Information is an answer to a question asked

The definition highlights two insights. One is the power of asking questions, because in most cases when you ask something it is something that bothers you, so the answer to the question is also an answer to a need.

The other insight is that in order to answer a question certain data is required, and through the question that data becomes information.

In order to manage an organization successfully questions have to be asked and each one of them is directed to highlight a required aspect for one of two categories of managerial needs:

1. Identifying new opportunities and how to draw the value from them
2. Identifying emerging threats and how they could be eliminated or controlled

The first category is about new initiatives for success. The second category is about protecting your back.  Both are critical to every organization.

Goldratt’s third question is:

What are the current usage rules, patterns and behaviors that bypass the limitation?

Without the means of gathering data from many sources the decision makers have to make decisions, the practice has to be based on the following elements:

• Using the routine data from the ERP or legacy system of the organization
• Using the intuition of the key people in the organization closest to the specific topic
• Employ a general ultra conservative approach, due to the unknowns and the perceived risk

The most important element is the use of intuition, based on one’s past experience.  So, it is certainly relevant data, but its quality is questionable.  The lack of objectivity, the various personal biases and being very slow to embrace any change, comprise the problematic side of intuition.

Intuition will still take a big role in the future. However, the ability provided by certain analysis based on data, unavailable before true big data, to check the validity of the initial intuition (especially the hidden assumptions) and also to be the source for new insights that could inspire new intuition could settle a new relationship between hard analysis and intuition.

TOC people argue that on top of intuition there should be cause-and-effect analysis that enables great managers to speculate right even when actual data is minimal. This is sometimes true, but as all cause-and-effect are based on observed effects, which are not always true facts, then even the most robust logic cannot deal with too many unknowns without data to rely on.

So, how could we improve our ability to spot new opportunities and emerging threats with the aid of the new IT capabilities of accessing huge amount of data?

The big trap of using the new IT capabilities is: losing the focus, investing huge amount of efforts on searching for data, analyzing it and eventually come up with almost nothing.  This is a real threat to many organizations.

The direction of solution we offer is building a high-level strategic process, run by a special team operating as a headquarter function that follows these steps:

1. Decide on a prioritized list of worthy objectives that are not satisfactorily achieved
2. For each of the objectives identify the key obstacle(s) and what is required to overcome them. We assume many of the obstacles are due to unknowns
3. Based on the above come up with a prioritized list of specific questions that require good answers, which currently are not available with a reasonably high confidence level.
4. Search for the specific data required for answering the questions. Many times the search is for external data, but then import the data to a central internal storage
5. Generating the global picture how to achieve more of the top objective. The answers to the questions are merged with cause-and-effect plus intuition to create possible alternatives for actions. The final analysis is submitted to the decision makers.

The above process is similar to what Intelligence Bureaus are doing for countries. The priorities and the means are clearly different.  Countries most critical questions are about threats, much less emphasis on opportunities, and their means to collect data are usually illegal with special permit from the government.

Customizing the process for true business intelligence isn’t trivial. The big mistake of imitation is ignoring the basic differences.  However, ignoring the similarities and the opportunity to learn from a well established process is another huge mistake.  Given the difference in ethics, priorities and means, the basic need and the analysis tools are similar enough, and the emergence of Big Data gives the potential value great chance of being materialized.

What makes these efforts worthy to go after is the simple fact that the underlying new insights do not clash with any deep paradigm of big companies.

We, Amir and I, will be glad to take part in such an endeavor. We have delivered a webinar on the topic that goes deeper into analyzing the value of Big Data.  The recording of our webinar on the topic can be viewed on TOCICO site, https://www.tocico.org/page/replay?.

In another post we intend to deal with the potential value of simulations to gain new insights and answering very troubling questions.  Like Big Data, and actually any new technology, simulations could bring huge value, but require special care from severe pitfalls.