Soon after my son introduced me to his new girlfriend, I sent her a Facebook “friend” invitation. Two minutes later, my son called to berate me for committing a grave social network faux pas, to which I responded, “Listen, I invented the Internet, and I’ll do whatever I darn well please there.” Although I cannot […]
read more »
Big Data is still an emerging field, with far too many products, promises and pratfalls. Despite all that, this is the season when industry pundits make fearless predictions about how Big Data will evolve in 2016. Many prognosticators take a high-risk approach, titillating the reader with highly speculative projections. If it comes true, the pundit […]
read more »
The Hadoop ecosystem has a well-known hole: there is no good tool for near-real time analytics on Big Data. For example, suppose you have 500 milliseconds to calculate the average temperature reading from a set of sensors, based on some ad hoc user-specified criteria such as location, time and sensor type. Pre-aggregation is not feasible, […]
read more »
Because I love my job, I cannot stop thinking about software engineering even when I am on vacation. My recent holiday in Rajasthan and Varanasi in Northern India provided an opportunity to truly admire some enterprising inventions of the ancient times that are now outdated. I believe this attitude of respectful admiration can help companies […]
read more »
When NoSQL databases first appeared, in response to the need to process high volumes of unstructured data, the “o” in NoSQL was small, and it meant that these systems had “No” SQL support – SQL was meant for traditional structured relational databases, not these new schema-less contraptions. But, developers know and love SQL, so these […]
read more »
In my position as Chief Data Officer for Ness Software Engineering Services, I talk with many companies about their Big Data challenges and needs. Some of these companies have chosen to analyze their Big Data to enable use cases such as online personalization or creating a 360-degree view of their users. Others have been forced […]
read more »
It seems Hadoop is the ‘flavor of the month’ in IT these days, and commands conflicting headlines month after month. What is Hadoop, and where did it come from? Like many great innovations, Hadoop was created to solve an entirely different problem – building a web search engine. In 1999, Doug Cutting developed Lucene, a […]
read more »
Last time we explored Massively Parallel Processing (MPP) databases as a way to run SQL queries on Big Data at Rest. One of the leading commercial products in this category is Pivotal’s Greenplum, which, at its peak, achieved annual revenues of over $100 million. The rise and fall of this product provides an illustrative lesson […]
read more »The first technology family we will explore for Big Data at Rest is Massively Parallel Processing (MPP). Traditional relational databases scale up by adding more server nodes, which share the same disks. Some databases, such as Oracle’s Real Application Cluster (RAC) go even further, and share the same in-memory cache across all servers. The benefit […]
read more »
The rubric of Big Data covers a lot of varied problems. Knowing how to classify a Big Data problem is the first essential step in proposing a solution. To illustrate, suppose your retail business has two critical needs: Firstly, you need to analyze historical sales data in your Enterprise Data Warehouse, going back 10 years, […]
read more »