The desire to make better decisions faster is one of the fundamental drivers of new big data analytics technologies and a general push toward data-driven decision-making. But the relationship between data and intuition -- the old 'gut feeling' -- is a complicated one, says Peter Swabey, senior editor, technology at the Economist Intelligence Unit (EIU), the research and analysis division of The Economist Group.
"They both play a role," Swabey says. "The process of developing data is the process of trying to identify what the true state is. In identifying that, your intuition could be a useful guide."
In an effort to better understand how business decisions are made, predictive analytics firm Applied Predictive Technologies (APT) asked the EIU to conduct a study, resulting in a report released this week: Decisive action: How businesses make decisions and how they could do it better.
In February 2014, EIU surveyed 174 senior managers and executives from around the world and from a range of industries, 49 percent of whom represented organizations with more than $500 million in annual revenue. EIU also conducted in-depth interviews with practitioners and experts.
The study found that 42 percent of respondents said they collect and analyze data as much as possible before making a decision, while an additional 17 percent said they approach decisions empirically by developing hypotheses and performing tests to prove or disprove them. In other words, 59 percent rely on data to help them make decisions. Of the remainder of respondents, 32 percent said they seek to collaborate on decisions as much as possible, while only 10 percent said they primarily rely on intuition to make decisions.
"Despite the apparent popularity of data-driven decision-making, however, intuition is in fact valued highly," says Jane Bird, author of the EIU report. "Nearly three-quarters of respondents (73 percent) say they trust their own intuition when it comes to decision-making. Even among the data-driven decision-makers, over two-thirds (68 percent) agree with that statement."
And, Bird notes, 68 percent of them would be trusted by their peers and superiors to make a decision not supported by data.
"However much data you take in, and whatever the interview process, when you get to the end there has to be an element of gut feel too," says Alison Robb, group director for people, customer, communications and commercial at Nationwide Building Society, one of the expert practitioners interviewed for the EIU report. "It's partly chemistry, experience and knowing what you do and don't like."
Intuition Can Help Identify Problems With Data Analysis
Perhaps nothing illustrates the inextricable link between data-driven decisions and human intuition better than how people would respond if the available data contradicted gut feeling. Fully 57 percent of survey respondents said that if confronted with that situation, the first thing they would do is reanalyze the data; 30 percent said they would collect more data and only 10 percent said they would take the course of action suggested by the data.
As an example, Rupert Naylor, vice president of APT, says that just last week, an APT client ran a test of the effectiveness of putting inserts into newspapers: Would they drive more sales?
"They ran the analysis and the results came back completely flat," Naylor says. "There was no difference between the performance of the tests and the control."
Surprised by the results, which on the face of it suggested that inserts don't drive additional sales at all, the client asked APT to check whether anything had gone wrong with the analytics software. But APT found the analysis had been run properly. Dumbfounded, the client spoke to the media agency responsible for the test, only to discover that the agency had failed to run it.
"That was a great victory for data," Naylor says. "The data was saying that nothing had happened."
"It's also a vindication of intuition," Swabey adds. "It's the fact that the result was counterintuitive that indicated something was wrong."
Even as organizations move to become more data-driven in their decision-making, Bird says the survey findings suggest that intuition should also be valued and cultivated.
"Evidently both intuition and analysis contribute to effective decision-making, in business as in life," she says. "Rather than a weakness that must be avoided, intuition should instead be seen as a skill that is appropriate in the right circumstances."
In fact, she suggests that to improve the nature of decision-making within organizations, those organizations must acknowledge that decision-making itself is a standalone skill that can be improved just like other skills. Furthermore, she says, organizations must hold executives accountable for the quality of their decisions.
In the report, 19 percent of respondents said decision-makers in their organizations are not held accountable at all, while 59 percent acknowledged that poor decision-makers will still be allowed to progress within their organization. And it may be that this state of affairs persists due to a lack of transparency in decision-making. Two-thirds (64 percent) of respondents said information about who makes decisions and why is restricted to sufficiently senior employees; only 26 percent said that information is freely available to everyone.
"This implies that in a significant minority of organizations, senior managers take decisions behind closed doors for opaque reasons and, should those decisions turn out to be the wrong ones, are not held accountable," Bird says. "This is a disturbing state of affairs."
Gerry Grimstone, chairman of insurer Standard Life, says it is essential to cultivate accountability while at the same time ensuring that executives don't exist in a climate of fear that prevents them from raising serious issues until it is too late.
"I'm a great believer in accountability, and I'm happy that my board members have got some liability," he says. "They take better decisions if they think that there are reputational consequences if they get it wrong."
Thor Olavsrud covers IT Security, Big Data, Open Source, Microsoft Tools and Servers for CIO.com. Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline, Facebook, Google + and LinkedIn.