These days I’m feeling more inspired to blog about the emerging “Big Data” technologies than about generic Product Management or Product Marketing.

I am naturally interested in this sector because half my career was spent in the Business Intelligence sector of Enterprise Software.  And I really liked it for a long time. But over the last few years, Business Intelligence had matured quite a bit: a lot of the products were becoming commodities, all the vendors’ messaging sounded essentially the same, and customers were increasingly price-driven.

So, as a product manager, the thrill of managing BI products was pretty much gone. Everyone held too many preconceptions and made too many assumptions about, well, everything: customers, their use cases, the competition, and more.  Hardly anything surprised.

And then the landscape shifted — away from the big BI and RDBMS monoliths and toward startups with awesome new “Big Data” technology.  Technology that could solve customer problems that were previously assumed to be unsolvable.  Technology that made a quantum shift in the experience users had with using business intelligence.

What is “Big Data”?##

“Big Data” is a kind of annoying umbrella term for the various technologies designed to help companies understand and analyze the HUGE amounts of data they are now collecting and that are available from public sources.

It encompasses:

  • New breeds of massively parallel databases that can be split across dozens of independent machines (aka MPP databases)
  • Hadoop and MapReduce
  • NoSQL databases that give up some aspect of their relational database ancestors (such as joins, SQL-like languages, and/or transactions) to distribute data and processing over huge clusters of low-end machines.
  • The next generation of visualization tools to help users make sense of massive amounts of data
  • Heavy-duty data analysis, including predictive analytics