Showing posts with label Search Based BI. Show all posts

Thursday, August 9, 2012

Explore More Data with Oracle Endeca Information Discovery

BI에 날개를 달아주자 .

Search Based BI with Endeca Information Discovery from Oracle

see video:
http://www.oracle.com/ocom/groups/systemobject/@mktg_admin/documents/webcontent/videoplayer-ocom.html?bctid=1545390829001&playerType=single-social&size=events

Product Infomation : http://www.oracle.com/technetwork/middleware/endeca/overview/index.html

source : http://www.youtube.com/playlist?list=PLF23635ACA47F1E6D&feature=plcp

Getting Started with Endeca Information Discovery v2.3

by Oracle Endeca Information Discovery Channel

Play all

10 likes, 0 dislikes

Monday, August 6, 2012

펌:The BI-Search Evolution

source: http://www.information-management.com/newsletters/business_intelligence_bi_search-10017332-1.html?zkPrintable=true

The BI-Search Evolution

By David Caruso

MAR 10, 2010 5:31am ET

Several years ago, I was involved in a consulting engagement with the IT operations of a multibillion dollar firm. The focus was on the areas of budgets, ROI achievement, effective system use and user satisfaction. While the IT staff did a good job of managing budgets, they often struggled to deliver strategic business benefits to the users. One of their biggest problems was the inability to deliver timely information to executives and business users who needed to make ad hoc decisions. Consequently, expensive MBA types scrambled to extract data from business intelligence reports and spreadsheets in order to prepare analyses for the managers.

A recent Forrester Research survey noted that when it came to BI environments, 59 percent of the respondents indicated that users were unable to access 100 percent of the data needed for reporting and analytic work. Additionally, 78 percent of the respondents indicated that their BI environment did not enable exploration and analysis with features such as adaptive data models, unlimited dimensionality and guided analysis.

Information access and exploration has become more challenging as more people attempt to make more decisions based on more data. The people who need answers can’t find what they need – nor can they easily use what they found. Companies have been implementing and using information systems for decades, so what makes this so difficult and expensive? The answer is apparent when you look at the underlying source systems and the process of bringing the data from those sources together:

•Too many disparate data sources. New data sources are being introduced every day, and existing sources undergo constant changes. This complicates the task of unifying data and allowing for decision-making based on the most up-to-date information.

•Evolving user needs. The information needs of users change as rapidly and continually as the business needs evolve. Users also have expanded data universes, moving beyond just spreadsheets to include all company-wide and even Web-based data.

•Expensive and time-consuming data modeling. Often, users don’t know what information they need, so it’s difficult for them to articulate what data they require for decision-making. Because IT attempts to anticipate all the answers a user will ultimately want, this process often requires extensive data modeling in order to get the right answers.

•Power tools for the everyday user. Many analytic tools are intended for sophisticated power users, but these users are only a small fraction of the decision-making population in a company. Today, almost every business user is expected to make informed decisions.

Fortunately, easy-to-use search and the power of BI are finally merging, so IT can now deliver on the promise of providing users with all the data necessary to make strategic business decisions, and the power to discover, explore and analyze.

Taken separately, traditional BI and enterprise search tools were each designed to solve problems other than ad hoc decision-making. BI was originally developed for reporting on structured data while search was designed for retrieving unstructured documents. As a result, each technology fall shorts in different ways in several key areas – a good user experience, the types of accessible information and the ability to respond to rapid change.

Because of its focus on reporting and structured data, BI tools are good at answering predictable questions and reporting on key performance indicators. However, they aren’t as effective at answering new or ad hoc questions, requiring the user to request custom reports and cubes from IT analysts.

This is because the rigid, hierarchical data models in BI tools only allow limited exploration and are often complicated to use. Even with rigorous data modeling, most BI tools cannot access unstructured content, and adding new data sources requires analysis and redesign of the data models as well as the reports, analytics and dashboards they drive.

On the other hand, everyone uses search today because, in many ways, it has become the simplest form of computing. Googling someone or using search on an e-commerce site or even on a company home page is the starting point of many users’ regular computer use. However, basic search, with its incomplete data model and document-centric retrieval, also allows for only limited exploration, depriving users of necessary context. Structured data is an afterthought. And, while new data sources can be easily added, providing context and exposing relationships to existing data is difficult.

However, BI and search can be combined to preserve the strengths of both and mitigate the drawbacks of each.

Enabling Discovery

To understand how combining search and BI can bring a richer solution to bear on business decisions, we have to first consider how humans actually make an ad hoc decision.

In daily decision-making, people formulate their next question based on the answer to a previous question. In the process, people often need help formulating good questions because they want to understand what the alternatives might be when making trade-offs. As they gain insight into their problem, they can use additional filters, graphs and visualizations to drill down and explore deeper.

In the business world today, people typically rely on BI systems to get an answer to a known business problem and rely on search to find information. Unfortunately, answers to either structured BI queries or text searches are only as good as the question posed. Although users might glean insight from the results returned in either case, they might never know if the question asked was the right one. Only a few business questions are simple enough to be served with a "hole-in-one" answer.

The convergence of BI and search technologies can enable a user to expose relationships in data that can often lead to an unanticipated answer or new revelation – without the necessity of the perfectly formed question.

Unification of the Data

First, we must note that BI is based on a schema-driven model. That schema holds the key to what can be searched or navigated. But BI usually doesn’t accommodate complex data or unstructured content. In addition, in order to navigate the data effectively, applications must be created specifically for the query at hand.

Data-driven exploration and query refinement, allowing for search on both structured and unstructured data, is important when the data is heterogeneous and hard to understand for users – such as when unstructured content is being included in the search process. This flexibility allows IT to unify heterogeneous, changing data and content from multiple sources without the headaches and expense of traditional data modeling. Likewise, it enables IT to incorporate data from any format, structured or unstructured, and makes it possible to navigate across unstructured documents by automatically extracting structure from them.

Now, because the data is self-describing, it is able to build a dynamic data model; in effect, automating the data modeling task. This enables faceted search and navigation (or guided navigation) which allows the user to elaborate a query progressively, seeing the effect of each choice in the results set. The real power in this kind of search is that someone can expose new data relationships that help drive an unanticipated answer without having created the perfect query. For example, guided navigation enables users to find products or categories via attributes such as part numbers and commodity groups, size and weight. This allows users to enter a few keywords to locate information. The returned information is organized by category (or dimension) and then, using a graphical interface, users navigate the data and its relationships to locate their necessary answer.

Benefits for Users and IT

With search capabilities opening up the data and bringing a new level of ease of use to the analytic power of BI, new benefits include:

•Users will be able to execute data queries via a search box using natural language. Users can start a decision process with a simple search such as “machined parts” and be able to refine the data on a point-and-click basis. This helps users who don’t know how to structure queries find all data available to them, because the faceted search will organize the dimensions and open up the data for ad hoc interaction.

•A single point of entry for multiple BI systems, operational data stores and additional content. A data-driven model simplifies the data preparation and opens up the universe of data that can be explored. Users will be able to combine and explore data from any system, making federated queries against multiple BI systems and underlying source systems easy. This enables users to go to a single entry point to access all the data and content they need, no matter where it may have originated.

•Access to all the data, regardless of type. The explosion of information sources means businesses can make better decisions. Users want to access all relevant data without regard for whether it’s structured, unstructured, new or old. By unifying structured data and unstructured content, and making it all searchable, a search-enabled BI environment allows access to more data more quickly, with more flexibility and less modeling or integration overhead.

Friday, August 3, 2012

"Googlizing" BI with Search-Based Applications

source : http://tdwi.org/articles/2011/06/08/googlizing-bi-with-search-based-applications.aspx

"Googlizing" BI with Search-Based Applications

Unstructured data holds essential business insights. How can you get to that insight?

June 8, 2011

By Eric Rogge, Sr. Director of Marketing, Exalead

Organizations are increasingly storing vast amounts of unstructured data in new Hadoop, NoSQL, and MPP analytic databases, and business intelligence tools are getting better about connecting with them.

Still, even with improving connections between BI and unstructured data stores, the challenge with today's business intelligence deployments is that they only enable quantitative analysis of a fraction of an enterprises' information assets. That's because the majority of information available to an enterprise is unstructured content held in documents, e-mail messages, collaboration forums, and on the Web. Enterprises now realize that to have a complete, 360-degree view of their operations, they need to analyze that unstructured data. That analysis involves both qualitative assessments as well as quantitative analytics. The challenge of BI isn't storing the unstructured data; it is the significant back-end development work needed to gather and quantify unstructured information sources.

Missing from an enterprise's portfolio of BI tools are search and semantic processing technology, which can efficiently process unstructured data into gists and metrics, plus handle large volumes of data from widely dispersed sources.

The effectiveness of today's BI solutions can be improved by working in conjunction with search-based applications (SBAs). SBAs are a new, emerging category of search and semantic technology that aim to improve operational productivity through processing, analysis, and delivery of key information drawn from internal and Web unstructured data. SBAs are a form of business intelligence and complement the highly quantitative analytics delivered by traditional BI products.

Search-based applications complement the ad hoc analysis and quantitative reporting typical of BI implementations. Where BI addresses the what questions, SBAs address the who, how, and why questions to give qualitative cause-and-effect explanations. They do this by collecting and co-displaying quantitative metrics and explanatory text in the same view. SBAs are also useful for extracting customer sentiment and other informational trends from the Internet -- a complex task beyond the capabilities of traditional BI.

By integrating semantic search-based applications with BI information sources (sometimes called the "Googlization of BI"), companies gain a broader understanding of their business activity that enables better business decisions to be made faster. Instead of using a single source of data as with traditional BI, SBAs can simultaneously access a wide variety of information sources while combining structured and unstructured data to provide a holistic, 360-degree view of the enterprise.

SBAs handle staggering amounts of data -- petabytes in some use cases -- while simultaneously providing Web-search-style, natural-language query interfaces that appeal to ordinary users. Today's workers, accustomed to fast and easy Google searches on the Web, can now gain the same easy-to-use tools to help them unlock information in the enterprise and gain insights for better decision making.

SBAs have a different information purpose than do BI applications. Whereas internal and external accounting standards demand focused, precise numeric precision in BI applications, many operational decisions require a broad perspective, sometimes using a collection or profile of facts such as dates, contacts, impending transactions, milestones, and opinions to provide a complete understanding. Now that audio and video are becoming common information delivery mediums, the ability to transform such multimedia files into text (and then into analytic data) is becoming important, perhaps even critical in some situations. Emerging technologies, such as voice-transcription software, are adding to the deluge of unstructured data in the enterprise, which continues to grow exponentially each year.

However, not all search-engine-derived technologies are equal. Companies looking to leverage the power of SBAs to improve BI should look for several capabilities in a solution. To most effectively boost BI, SBAs must be able to structure unstructured data (not simply index it), as well as integrate that information into the corporation's existing structured datastores. The key to this ability is semantic search technology, which analyzes the content of unstructured data to make sense of the information and rapidly identify relevant data.

In addition, companies should look for SBAs that feature service-oriented architectures (SOAs) to integrate decision tools for each user, enabling rapid deployment and simple integration within the company's information ecosystem. Effective SBA solutions for BI will also include faceted navigation, as well as the robust data security required in today's corporate environments.

SBAs combine the best of BI and enterprise search to deliver what until today has been an elusive goal for the enterprise -- that is, real-time BI with a comprehensive view of all information sources: structured and unstructured, internal and external. SBA-powered BI offers improved data scope and relevance by making better use of existing structured data and by exploiting new data channels ripe with essential information, such as e-mail messages, Office documents, and PDF files -- the vast amount unstructured data that, until now, was beyond the capabilities of BI.

As unstructured corporate data continues to grow exponentially, traditional BI will be left further behind. The efficient scalability of SBA ensures that corporations will be able to continue leveraging their growing stores of information in order to make business decisions more intelligently and more quickly.

Search-Based BI – the Next Innovation

source: http://www.information-management.com/newsletters/business_intelligence_search_analytics_data_warehouse-10020252-1.html

Search-Based BI – the Next Innovation
By Christian Becht and Marcel de Grauw

MAY 3, 2011 6:20am ET

Print Reprints (http://license.icopyright.net/3.7732?icx_id=10020252) Email inShare.1Since the 1970s, companies have used business intelligence to harness data, but these traditional BI tools weren’t built with today’s 24x7 economy in mind. But the rapid increase in data and online sources means that companies in today's 24x7 economy are faced with the challenges of requiring quicker, more user-friendly and flexible tools to cope with continuously evolving data.

Current BI offerings are evolving toward search-based BI in the future, where BI tools will integrate non-structured and external information in the same way Google indexes billions of documents daily while providing access to millions of simultaneous users.

The Early Days

The evolution of BI in large organizations goes back to the 1970s. In an increasingly competitive and global environment, business managers were looking for tools to support their decision-making processes. These early BI tools were focused on extracting data from source systems and on delivering reports displaying performance indicators; most of the time they were custom-made applications developed by internal IT specialists.

To satisfy the needs of a growing number of business managers, specific queries were integrated in the overnight batch and launched against the production systems. The objective was to get business information out of the production systems in the form of fixed-format standard reports, the so-called “print-outs.” On a regular basis, printed information was manually aggregated and keyed into presentation templates and data sheets. Some years later, the concept of the “information support database” was introduced to offload querying on the transaction systems and to improve the performance of the overall solution.

In response to the growing need for management support and reporting tools, software vendors like Pilot Software, Information Resources and Comshare jumped at the opportunity. The first generation of BI tools is often identified with the term executive information systems. The early BI tools included extract, transform and load capabilities, merged data from multiple sources, used relational databases, including what we later called star schemas, and built cubes for fast data retrieval.

The Second Wave in BI: Data Warehousing

In the early 1990s, the EIS pioneers fell on hard times. The costs of implementing corporate EIS systems were too high, and the required technical infrastructure wasn’t there, so the EIS tools had to include their own. In addition, EIS didn’t target and serve enough end users because of the “executive” connotations. At the same time, new innovations like data warehousing and online analytical processing (OLAP) began broadening the realm of decision support and initiated a larger category of BI tools. The so-called “data warehousing” model was further popularized as a means to describe a new set of concepts and methods to improve decision-making by using fact-based decision support systems.

During the second wave of innovation in BI, the production of management information was being industrialized by means of sequentially scheduled batch processes (information logistics). The entire production process, from the extraction of source data to the generation of reports, was being automated by means of specialized BI tools. The data warehousing model, as introduced in the early 1990s, has shaped the BI landscape ever since. Today the traditional BI model is still the guiding principle for designing new BI architectures in large organizations.

The established data warehousing model is being challenged by new concepts and technologies. Modern business managers are pointing to the shortcomings and drawbacks of the current model, both from an organizational and structural point of view. In other words: the data warehousing model as we know it has become too complex and expensive to maintain, and too rigid to provide the required speed of decision needed in today’s 24x7 economy.

Developing a traditional multilayered BI system is an expensive and labor-intensive exercise. To design and build interfaces, ETL jobs, star schemas, data marts and reports takes a lot of time. In addition, highly qualified experts from various disciplines are required to deliver and build a new version on time. Delivery cycles ranging from six to 12 months are typical because of the various teams and tools involved.

The Next Wave in BI: Information Intelligence

To fulfill its promise and to respond to future requirements, BI needs to become more intelligent, user-friendly and flexible. Today’s BI, based on the data warehousing model, is lacking some very basic features and functionality. Adding another BI tool will only increase complexity and costs and is, therefore, not an appropriate solution. We need to reconsider the basics of the current model and identify areas and technologies with the potential to improve things structurally. Areas to be improved include:

•Predictive analytics,

•Proactive alerts and notifications,

•Event-driven/real-time access to information,

•Accelerated integration of structured or nonstructured new data, either internal or external to the organization,

•Enterprise integration/closed-loop BI,

•Portal integration/mobile/ubiquitous access,

•Improved visualization/rich interfaces to empower business users,

•Management automation/decision engines, and

•Collaborative tools to leverage collective intelligence.

The BI of the future is becoming the brain and the central nervous system of organizations. Management information doesn’t find itself locked in a data mart or in a management report anymore; instead, it is automatically being reinjected in operational source systems to adapt to ever-changing market conditions. The next wave in BI, information intelligence, will be the lifeblood of organizations.

Information Intelligence, or the intelligent use of information, extends BI beyond the traditional data warehouse and query tools to include automated decision-making and real-time/event-driven technologies. Information intelligence is about building smarter business processes and making BI more user driven and flexible.

One of the technologies we believe is capable of transforming future BI architectures is enterprise search engines. Enterprise search engines have the capacity to simplify and improve BI in large organizations. This is because search engines possess the following attributes:

•Flexibility – search engines can handle both structured and unstructured information in various formats.

•The ability to cope with continuously evolving data structures. (Indexing both existing and new data does not require extensive data modeling. This is in contrast with the modeling of the data warehouse, which is time-consuming not only when the model is created, but each time new data is added to the data warehouse.)

•Search engines enable content-driven dimensional navigation. At each step of navigation, search engines propose different possibilities to filter results according to the content of the data sets that are being analyzed in near real time. This feature makes the traditional approach based on predefined data cubes obsolete.

•They are able to analyze data without the need to know the various data types, unlike solutions based on relational database management systems.

•Search engines can work with existing information systems (e.g., data warehouses, data marts, production systems, etc.) and are able to provide a federated view of data with the required level of performance, in contrast to federation approaches based on RDBMS that fail to address performance requirements. At the same time, the federated business view can encompass new data sources and provide cross-domain data navigation.

•Search engines utilize a familiar Google-style interface which empowers business users to retrieve data in a way that matches their questions rather than in a prestructured way that often doesn’t suit their real business needs

•They can fill the gaps in traditional data warehouse architectures when external and unstructured data is needed to support decision-making.

•Search engines include functionality to automatically generate categories and clusters, hence improving the contextualization and meaning of data.

•They aggregate and analyze data, in addition to enabling end users to expose relationships and to find patterns in data without the necessity of the perfectly formulated question or query. Search engines provide a powerful complement or alternative to SQL language that remains at the heart of today’s BI solutions - even though it was created more than 35 years ago.

Toward a Search-Based BI

Based on technologies like Exalead, Autonomy and Fast, billions of documents are being indexed on a daily basis from multiples source systems like enterprise content management, enterprise resource planning, customer relationship management, data warehouses and other legacy systems.

Information is being collected in near real-time and presented to end users through user-friendly interfaces that can be extended using the powerful rich Internet application standards. Because of the nature of enterprise search engines the time required to implement a search-based BI solution is heavily reduced compared to the time that is needed to design and build a traditional BI system. Furthermore, performance is not an issue in search-based BI, neither in terms of number of users nor in volume of data.

Future BI systems, integrating nonstructured and external information, will benefit from the proven scalability features of search engines. Search-based BI is leveraging and not replacing investments in existing BI systems and is capable of getting the long-awaited business benefits out of the investments in existing data warehousing environments.

While search-based BI won’t replace current BI systems in the short-term, search-based applications are being used as a complement to cater for the shortcomings of existing BI systems - such as the ability to answer critical business questions more rapidly and cost effectively. In the long run we will see search-based solutions transforming the BI domain because of its inherent features. The combination of BI and search-based solutions will preserve the strengths of both and mitigate the drawbacks of each.

Christian Becht leads the Business Information Management practice of Capgemini in France. He spent the last 12 years creating and growing the Business Intelligence and Enterprise Content Management practice in France, now with up to 700 consultants. Christian is a telecoms engineer who graduated in 1986 from the French “Ecole Nationale Superieure des Telecommunications de Bretagne”. He began his career in the telecom sector where he designed IT solutions able to handle huge amounts of data in a cost effective way. For more information you can join him at christian.becht@capgemini.com.

Marcel de Grauw MSc. (1970) is a Managing Consultant at Capgemini in Paris. He has been active in Business Intelligence since the early 90s. Before joining Capgemini in 2001, Marcel worked for Philips/Origin in The Netherlands. He is currently working on next generation BI architectures based on Search engines and AI. Marcel holds a Master degree in Business Administration from the Erasmus University in Rotterdam. For more information, you can join him at marcel.de-grauw@capgemini.com

나의 인생 이야기