Skip to main content

Getting Control of BIG DATA


I got a copy of HBR’s October edition and like its featured article “Getting Control of BIG DATA”. A is fantastic article giving deep insight in the subject! ...and the credit goes to authors: Andrew, a research scientist and Erik, an MIT professor. 

Big data is definitely having huge business potential but at the same time poses few business challenges especially from corporate culture's perspective.

Here are few bullets of general interest:

  • Why Big Data? …analyzing it leads to better predictions and better predictions yields better decisions. In sector after sector, companies that figure out how to combine domain expertise with data science will pull away from their rivals.
  • Web Fact: 2.5 exabytes of data created daily and this number is doubling each 40 months
  • Retail fact: Walmart collects 2.5 petabytes of data every hour from its customer transactions
  • Case Study (Retail) - Sears: Sears is using Hadoop cluster for data analytics. (personalized promotion time came down from 8 weeks to 1 week, saving cost and time!) It is interesting to know that they are directly storing the data on Hadoop clusters and doing real time analysis.
  • Case Study (Aviation) - PASSUR : Improved airline ETAs which helped in improving staff efficiency whose worth would be several million dollars a year at each airport
  • Big data can shift the corporate culture by muting the HiPPOs (Highest Paid People's Opinion): Rather than relying on “intuitive” decisions from people high up in the organization, it would be based on real data analytics.
  • It would redefine domain expert: As the big data movement advances, value (of role) of domain experts will shift from their HiPPO style answers to their data specific questions.
  • New lucrative role of Data Scientist: a person who can extract treasure out of messy, unstructured data

The article contains in depth discussion around this and it is worth to grab this copy from the stall and churn it out if you are serious business in upcoming technologies.

Comments

Post a Comment

Popular posts from this blog

Mastering Hadoop: Book Review

I came across a book Mastering Hadoop published by Packt and authored by Sandeep Karath. Here is my detail review about the book- SUMMARY This book is based on most popular massive parallel programming (MPP) framework " Hadoop " and its eco-system. This is an intermediate level book where author goes in depth on not only the principle subject but also on most of the supporting eco-systems like hive, pig, stream, etc. The book has 374 pages with 12 chapters, the ToC  itself is spanned across 7 pages! It has conceptual as well as hands on lab experiences with lot of code churned into. OPINION The book starts with genealogy of Hadoop where the author has nicely narrated the evolution of web search to current state and then various releases of Hadoop. Good reasoning as why Hadoop 2.0 was essential to move ahead from previous version. Touches the architecture starting from high level 3-layered, drilling down step by step to cluster and node level. Describes all the feat
129 comments
Read more

Hadoop Ecosystem

When it comes to Hadoop, still some people believe it as a single out of box system catering all big data problems. Unless you are thinking of some third party commercial distribution, this is not correct. In reality, Hadoop on its own is just HDFS and MapReduce . But if you want production ready Hadoop system, then you will have to also consider Hadoop friends (or components) which makes it a complete big data solution.  Most of the components are coming as apache projects but few of them are non-apache open source or even commercial in some cases. This eco system is continuously evolving with large number of open source contributors. As shown in the above diagram. The following diagram gives high level overview of hadoop ecosystem. Figure 1: Hadoop Ecosystem The Hadoop ecosystem is logically divided into five layers which are self-explanatory. Some of the ecosystem components are explained below: Data Storage is where the raw data will be residing at. There are mul
72 comments
Read more

Is blockchain a technology or an algorithm?

After a phenomenal growth of bitcoin in 2017, all of sudden everyone in the cyber world has started talking about crypto-currency and the technology behind it - Blockchain . I am sure you too must be flowing through this new fanfare. So here I would be trying to explain this platform in just 11 mins . What is Blockchain? Blockchain is an ever-growing list of transactions , called Blocks which are always linked to their previous Blocks and are secured by cryptography hash. This Blockchain will be stored on distributed peer-to-peer (P2P) network nodes. If you are familiar with BitTorrent, you can easily understand this P2P communication. Each block has three things: data, its own hash and hash to previous block Data can be regarded as a ledger. Hash can be compared with fingerprint, which is a unique identification of the block. It is generated based on the content and even a single character change will make it different. Hash to previous Block: this creates a link
39 comments
Read more