Big Data – Fast Analytix

Data has the potential to profoundly affect the way we do business.

The quote on CNBC really exemplifies this – “Data is the new Oil.”

Data is a natural resource that is growing bigger. Like any resource, it is difficult to extract. It comes in many types – or a huge variety. It is also difficult to refine, or analyze. Many organizations do not even tap into this natural resource – they ignore data, or they use it for just one purpose. This is largely because it is difficult to structure and restructure for different purposes. But some organizations have cracked the code, and they have figured out how to process and analyze data available to them, and they are utilizing it to achieve breakthrough outcomes.

If data is a natural resource, what is your company doing to capitalize on it?

In the new era of computing, the growing complexity of new sources of data and a variety of data types creates challenges. Left unchecked, data is becoming more uncertain and directly impacts the initiatives it supports such as analytics.

Big data has become a big deal, and it’s causing many organizations to ask, “Does big data equal big value for my business—or is it mostly big hype?” “What differentiates the big data programs that generate value from those that do not?” and “Just how much value can big data achieve?” The answers to these questions might surprise you.

The key aspects of a Big Data platform are:

1. Integration — the point is to have one platform to manage all of the data. There’s no point in having separate silos of data, each creating separate silos of insight. From the customer pointof- view (a solution POV) big data has to be bigger than just one technology

2. Analytics — a very important point. We see big data as a viable place to analyze and store data. New technology is not just a pre-processor to get data into a structured data warehouse (DW) for analysis. The game has changed – unlike DBs/SQL, the market is asking who gets the better answer and therefore sophistication and accuracy of the analytics matters.

3. Visualization — need to bring big data to the users — for example, a spreadsheet metaphor is well known technological approach understood by business users.

4. Development — need sophisticated development tools for the engines and across them to enable the market to develop analytic applications.

5. Workload optimization — improvements upon open source for efficient processing and storage

6. Security and Governance — many are rushing into big data like the wild west. But there is sensitive data that needs to be protected, retention policies need to be determined — all of the maturity of governance for the structured world can benefit the big data world.

Profit from Big Data as a Business

Data is multiplying at an exponential rate, generated by sensors, social media, transactions, smartphones, and a multitude of other sources. Naturally, companies want to tap into the potential of these vast, fast-moving, complex streams of data to achieve step-change improvements in performance. But they should set their sights higher. Big data, as a business in itself, could create billions of dollars in additional revenues that can go toward fueling growth.

Companies in a variety of information-rich industries are already generating entirely new revenue streams, business units, and stand-alone businesses out of the data they hold. Over the long term, there is strong potential for such data businesses to appear in even the most traditional industries.

Information is multiplying inside businesses at an exponential rate, generated by sensors, social media, transactions, smartphones, and other sources. Companies increasingly want to tap into the potential of these vast, fast-moving, complex streams of data to achieve step-change improvements in performance.

Making Sense of Big Data

Big data, by itself, can’t change the world. But by applying the insights gleaned through the analysis of big data, companies can transform the way the world does business. Such insights can help reinvent different industries in different ways.

The main sources for big data are

1) Transaction data is EXISTING DATA that customers say is the main source of big data and they want to query/report on it. In an RDBMS used for transactions, every time you update/delete/insert and select (queries), this information is logged. The business user can analyze this data for business purposes, not just for recovery purposes: thus, what is this information used for – query and reporting.

2) Log data or machine data
This is web log, database log, sensor data that can be helpful to analyze use and inefficiencies.

3) Events

4) Social Media : Facebook, Twitter, Instagram, etc.

Also, using Still images/Video and Audio are not yet important generally as a BigData resource.

The final question is why do you want to deal with more data?
Because you want new insight This new insight is not only for top level executives This new insight will be used to get people throughout the enterprise to run the business better and to provide better service to customers.

We should also note that Big Data is not just Hadoop. Big Data is a platform.

Data warehouses are part of a big data platform that you may already be familiar with. They deliver deep insight with advanced in-database analytics & operational analytics. Data warehouses provide online analytic processing (or OLAP).
Stream computing is used to analyze streaming data and large data bursts for real-time insights.
And, of course, Hadoop provides cost-effectively analysis of petabytes of unstructured and structured data.