Search-Based BI – the Next Innovation

source: http://www.information-management.com/newsletters/business_intelligence_search_analytics_data_warehouse-10020252-1.html

Search-Based BI – the Next Innovation
By Christian Becht and Marcel de Grauw


MAY 3, 2011 6:20am ET

Print Reprints (http://license.icopyright.net/3.7732?icx_id=10020252) Email inShare.1Since the 1970s, companies have used business intelligence to harness data, but these traditional BI tools weren’t built with today’s 24x7 economy in mind. But the rapid increase in data and online sources means that companies in today's 24x7 economy are faced with the challenges of requiring quicker, more user-friendly and flexible tools to cope with continuously evolving data.

Current BI offerings are evolving toward search-based BI in the future, where BI tools will integrate non-structured and external information in the same way Google indexes billions of documents daily while providing access to millions of simultaneous users.

The Early Days

The evolution of BI in large organizations goes back to the 1970s. In an increasingly competitive and global environment, business managers were looking for tools to support their decision-making processes. These early BI tools were focused on extracting data from source systems and on delivering reports displaying performance indicators; most of the time they were custom-made applications developed by internal IT specialists.



To satisfy the needs of a growing number of business managers, specific queries were integrated in the overnight batch and launched against the production systems. The objective was to get business information out of the production systems in the form of fixed-format standard reports, the so-called “print-outs.” On a regular basis, printed information was manually aggregated and keyed into presentation templates and data sheets. Some years later, the concept of the “information support database” was introduced to offload querying on the transaction systems and to improve the performance of the overall solution.



In response to the growing need for management support and reporting tools, software vendors like Pilot Software, Information Resources and Comshare jumped at the opportunity. The first generation of BI tools is often identified with the term executive information systems. The early BI tools included extract, transform and load capabilities, merged data from multiple sources, used relational databases, including what we later called star schemas, and built cubes for fast data retrieval.



The Second Wave in BI: Data Warehousing

In the early 1990s, the EIS pioneers fell on hard times. The costs of implementing corporate EIS systems were too high, and the required technical infrastructure wasn’t there, so the EIS tools had to include their own. In addition, EIS didn’t target and serve enough end users because of the “executive” connotations. At the same time, new innovations like data warehousing and online analytical processing (OLAP) began broadening the realm of decision support and initiated a larger category of BI tools. The so-called “data warehousing” model was further popularized as a means to describe a new set of concepts and methods to improve decision-making by using fact-based decision support systems.



During the second wave of innovation in BI, the production of management information was being industrialized by means of sequentially scheduled batch processes (information logistics). The entire production process, from the extraction of source data to the generation of reports, was being automated by means of specialized BI tools. The data warehousing model, as introduced in the early 1990s, has shaped the BI landscape ever since. Today the traditional BI model is still the guiding principle for designing new BI architectures in large organizations.



The established data warehousing model is being challenged by new concepts and technologies. Modern business managers are pointing to the shortcomings and drawbacks of the current model, both from an organizational and structural point of view. In other words: the data warehousing model as we know it has become too complex and expensive to maintain, and too rigid to provide the required speed of decision needed in today’s 24x7 economy.



Developing a traditional multilayered BI system is an expensive and labor-intensive exercise. To design and build interfaces, ETL jobs, star schemas, data marts and reports takes a lot of time. In addition, highly qualified experts from various disciplines are required to deliver and build a new version on time. Delivery cycles ranging from six to 12 months are typical because of the various teams and tools involved.



The Next Wave in BI: Information Intelligence

To fulfill its promise and to respond to future requirements, BI needs to become more intelligent, user-friendly and flexible. Today’s BI, based on the data warehousing model, is lacking some very basic features and functionality. Adding another BI tool will only increase complexity and costs and is, therefore, not an appropriate solution. We need to reconsider the basics of the current model and identify areas and technologies with the potential to improve things structurally. Areas to be improved include:



•Predictive analytics,

•Proactive alerts and notifications,

•Event-driven/real-time access to information,

•Accelerated integration of structured or nonstructured new data, either internal or external to the organization,

•Enterprise integration/closed-loop BI,

•Portal integration/mobile/ubiquitous access,

•Improved visualization/rich interfaces to empower business users,

•Management automation/decision engines, and

•Collaborative tools to leverage collective intelligence.

The BI of the future is becoming the brain and the central nervous system of organizations. Management information doesn’t find itself locked in a data mart or in a management report anymore; instead, it is automatically being reinjected in operational source systems to adapt to ever-changing market conditions. The next wave in BI, information intelligence, will be the lifeblood of organizations.



Information Intelligence, or the intelligent use of information, extends BI beyond the traditional data warehouse and query tools to include automated decision-making and real-time/event-driven technologies. Information intelligence is about building smarter business processes and making BI more user driven and flexible.



One of the technologies we believe is capable of transforming future BI architectures is enterprise search engines. Enterprise search engines have the capacity to simplify and improve BI in large organizations. This is because search engines possess the following attributes:



•Flexibility – search engines can handle both structured and unstructured information in various formats.

•The ability to cope with continuously evolving data structures. (Indexing both existing and new data does not require extensive data modeling. This is in contrast with the modeling of the data warehouse, which is time-consuming not only when the model is created, but each time new data is added to the data warehouse.)

•Search engines enable content-driven dimensional navigation. At each step of navigation, search engines propose different possibilities to filter results according to the content of the data sets that are being analyzed in near real time. This feature makes the traditional approach based on predefined data cubes obsolete.

•They are able to analyze data without the need to know the various data types, unlike solutions based on relational database management systems.

•Search engines can work with existing information systems (e.g., data warehouses, data marts, production systems, etc.) and are able to provide a federated view of data with the required level of performance, in contrast to federation approaches based on RDBMS that fail to address performance requirements. At the same time, the federated business view can encompass new data sources and provide cross-domain data navigation.

•Search engines utilize a familiar Google-style interface which empowers business users to retrieve data in a way that matches their questions rather than in a prestructured way that often doesn’t suit their real business needs

•They can fill the gaps in traditional data warehouse architectures when external and unstructured data is needed to support decision-making.

•Search engines include functionality to automatically generate categories and clusters, hence improving the contextualization and meaning of data.

•They aggregate and analyze data, in addition to enabling end users to expose relationships and to find patterns in data without the necessity of the perfectly formulated question or query. Search engines provide a powerful complement or alternative to SQL language that remains at the heart of today’s BI solutions - even though it was created more than 35 years ago.





Toward a Search-Based BI

Based on technologies like Exalead, Autonomy and Fast, billions of documents are being indexed on a daily basis from multiples source systems like enterprise content management, enterprise resource planning, customer relationship management, data warehouses and other legacy systems.



Information is being collected in near real-time and presented to end users through user-friendly interfaces that can be extended using the powerful rich Internet application standards. Because of the nature of enterprise search engines the time required to implement a search-based BI solution is heavily reduced compared to the time that is needed to design and build a traditional BI system. Furthermore, performance is not an issue in search-based BI, neither in terms of number of users nor in volume of data.



Future BI systems, integrating nonstructured and external information, will benefit from the proven scalability features of search engines. Search-based BI is leveraging and not replacing investments in existing BI systems and is capable of getting the long-awaited business benefits out of the investments in existing data warehousing environments.



While search-based BI won’t replace current BI systems in the short-term, search-based applications are being used as a complement to cater for the shortcomings of existing BI systems - such as the ability to answer critical business questions more rapidly and cost effectively. In the long run we will see search-based solutions transforming the BI domain because of its inherent features. The combination of BI and search-based solutions will preserve the strengths of both and mitigate the drawbacks of each.



Christian Becht leads the Business Information Management practice of Capgemini in France. He spent the last 12 years creating and growing the Business Intelligence and Enterprise Content Management practice in France, now with up to 700 consultants. Christian is a telecoms engineer who graduated in 1986 from the French “Ecole Nationale Superieure des Telecommunications de Bretagne”. He began his career in the telecom sector where he designed IT solutions able to handle huge amounts of data in a cost effective way. For more information you can join him at christian.becht@capgemini.com.



Marcel de Grauw MSc. (1970) is a Managing Consultant at Capgemini in Paris. He has been active in Business Intelligence since the early 90s. Before joining Capgemini in 2001, Marcel worked for Philips/Origin in The Netherlands. He is currently working on next generation BI architectures based on Search engines and AI. Marcel holds a Master degree in Business Administration from the Erasmus University in Rotterdam. For more information, you can join him at marcel.de-grauw@capgemini.com