Big Data Analytics 101: How Is Big Data Different?
A Big Data problem (as opposed to a classic data problem) is best characterized by the 3 V’s:
- Volume: When the amount of data to be analyzed grows beyond 5 TB of storage, traditional database solutions such as Oracle or Microsoft SQL Server start getting prohibitively expensive and very hard to maintain. When the data gets beyond 100 TB, those database products can stop working entirely.
- Velocity: A traditional relational database can ingest around 1,000 rows per second per server. Netflix, which records very detailed information about the behavior of its users, needs to write 1 million rows per second. Clearly, some new technology is required.
- Variety: A classic relational database assumes that the data is structured, i.e., each row in a table has values for the same columns. For example, the fields required for employee data (e.g., department, manager, salary) are the same for all employees. But, some data is not so structured, e.g., keywords for a set of documents. Each document has its own nearly unique set of keywords, so defining a column for each potential keyword would be futile. Again, some new technology is required.
There are many other V’s that characterize Big Data, e.g., veracity, value, validity, volatility. But, these 3 V’s (volume, velocity, variety) are the core criteria we use to define whether a given problem is a Big Data problem or not.
Of course, there is no free lunch. In order to achieve higher volume, velocity and variety, Big Data technologies have to give up on other features that we take for granted in traditional relational databases, for example:
- Transactions: Suppose I want to transfer money to your bank account. There are two database operations that must occur: the sum is deducted from my account, and the sum is added to your account. These two operations must be performed as a single transaction – either both actions occur, or no actions occur. Many Big Data technologies cannot support this efficiently.
- Immediate consistency: If I write a record to Oracle and then immediately try to read it, I expect to find it. Not so for some Big Data technologies.
- Rapid JOINs: A relational database is normalized, i.e., it does not contain repeated data. For example, suppose we want to generate a report of books borrowed by library users, where each row includes the name, address and phone number of the user. It would be inefficient to repeat this user information in the database for every borrowed book. Instead, we would store the user information in a separate table, and enrich user information for each borrowed book via a JOIN on the unique user ID. Traditional relational databases are tuned to perform these JOINs at high speed, via indexes. Many Big Data technologies are bad at indexing and JOINs, and instead encourage duplicating data.
There are fundamental differences between traditional database products and Big Data systems. A traditional database is like a Swiss Army knife – it can adequately support a wide range of use cases, some better and some worse, but all are supported. A Big Data database is like an apple corer – it is exquisitely tuned for some very specific use cases, and will not support other use cases at all. That’s why it is vital to build a Proof of Concept around whatever Big Data technology you choose, to verify that it can handle your specific usage pattern.
Coming up in my next blog post, I’ll propose a 30,000 foot view of the Big Data landscape by classifying Big Data technologies into product categories.
