Cascading Lingual Makes it Easy to Migrate Enterprise Data Warehouse (EDW) to a Hadoop Data Lake
In the world of Big Data, companies are increasingly leveraging Cascading Lingual to facilitate the adoption of enterprise Hadoop because it makes it easy for SQL users to utilize their existing skills to quickly create and run applications on Hadoop. Bhushan Pargaonkar, Lead – Technical, explores Lingual’s heritage and the reasons that make it the best fit for most of the Hadoop SQL needs.
Lingual was created in collaboration by the developers of Cascading and Optiq, and it relies on the robustness of both.
- Cascading is the de facto Java API for creating complex data processing workloads and it is the engine underneath Scalding, Cascalog, and others. Cascading enables the developer to build Big Data applications once and switch to different, distributed execution environments without a significant code change. The same Cascading code can be executed on various run times with virtually no code changes. (As of today, Cascading supports Apache Flink and Apache Tez (in WIP-2.0 branch), and Apache Spark support is on roadmap.) Hence, the developer is able to secure the company’s investment in application development, and also leverage new innovations shaping the world of Big Data. Cascading is essentially a domain-specific language (DSL) for Extract-Transform-Load (ETL) on Hadoop, and it generates MapReduce jobs under the hood for data processing. Cascading supports a large number of relational data sources such as Derby, H2, Oracle, MySQL, PostgreSQL, Redshift, Teradata etc. It also supports NoSQL data sources such as Accumulo, Cassandra, HBase, Hive, and Solr.
- Optiq is a SQL parser and optimizer written by the author of Mondrian. Optiq (now Apache Calcite) is a highly customizable engine for parsing and planning queries on data in a wide variety of formats. It allows database-like access, and in particular a SQL interface and advanced query optimization, for data not residing in a traditional database.
Background of Cascading-Lingual:
Cascading Lingual is built on top of Cascading. This simplifies application development and integration by providing an ANSI SQL interface for Apache Hadoop. Going beyond ‘SQL-like’, Lingual is a mature implementation of ANSI/ISO standard SQL-92, built on top of the Cascading framework.
Databases were traditionally engineered in a monolithic stack, providing a data storage format, data processing algorithms, query parser, query planner, built-in functions, metadata repository and connectivity layer. Database developers typically innovate in some areas, but rarely in all.
Hadoop and NoSQL database systems decompose that stack into separate components, separating data, processing engine, metadata, and query language support. These components are heterogeneous with data in multiple locations and formats. The same is done within Lingual as well. Lingual brings together the mature query planning and optimization capabilities of Optiq and advanced data processing capabilities and extensibilities of the Cascading Framework. Lingual has rich ETL support which is built on top of Cascading. In simple terms the decomposition of Lingual looks like below:
The technologies are heterogeneous and independent of each other and work in conjunction to bring up an ANSI SQL interface to Hadoop and other data sources.
Lingual allows re-organization of the data between heterogeneous data sources and systems without affecting applications already built on that data. The Cascading Framework connects those components (data formats, engines, algorithms) without introducing unnecessary coupling or performance trade-offs.
Ten reasons why you should consider Lingual for Hadoop SQL needs:
- ANSI standard SQL parser and optimizer build on top of Cascading framework
- Relational Catalog view into large-scale unstructured data
- SQL shell to test and submit queries to Hadoop. Also works well with Squirrel SQL as a desktop client.
- JDBC driver to connect existing BI tools and critical applications to Hadoop
- Optimize computing costs by migrating SQL queries to Hadoop
- Lingual inherits the power of Cascading, which means it can now integrate and process data from heterogeneous sources with one SQL statement over JDBC
- Accelerate application development and testing
- Lower the Hadoop adoption barrier
- Unified SQL technology across all layers of Hadoop Data Lake
- Single platform for code development that evolves with changing Big Data landscape. Cascading applications can be easily ported from one execution engine to another without significant code change. This enables the adoption of the latest advancements and innovations happening in the Big Data industry. Other Hadoop SQL technologies, are tied to one specific implementation. Cascading 3.0 WIP now supports Apache Tez, and hence Lingual WIP 2.0 also supports Apache Tez. Applications can be developed on Hadoop and Cascading Jobs can be later deployed using Apache Tez as runtime with very little or no code change. Lingual thus can reap the benefits of Tez as Tez support matures
Why migrate to Hadoop Data Lake from EDW?
In today’s Business Intelligence (BI) scenario most organisations already have BI infrastructure built using traditional EDW technologies. Those implementations primarily use SQL as language for defining ETL operations and have tons of such ETL jobs already developed and maintained. The organizations however face issues with their EDW infrastructure in terms of storage and Total Cost of Ownership (TCO) for building and maintaining analytics implementations. Hence many organizations are increasingly offloading part of EDW to Hadoop Data Lake. But building ETL in Hadoop has not been easy historically and requires niche skills in MapReduce development. Hence, many organizations have been working on building SQL or SQL-Like interfaces on Hadoop for easy development and migration to Hadoop. In such situation, Lingual is great fit as it is ANSI SQL-92 standard complaint and has JDBC driver implementation for easy connectivity from existing applications and BI tools. This makes Lingual a great contender for SQL technology of choice as most of the existing SQL queries can be reused using Lingual and Cascading on Hadoop.
Tips on EDW to Hadoop Data Lake migration:
- Don’t reinvent the wheel: Do not try to remodel and recreate ETL. This would take a significant amount of time and involvement from IT and business and will adversely affect the ROI on migration project. The best approach in such cases is “lift and shift” – use the current model and ETL flows, and try to migrate them as-is to Hadoop using Cascading and Lingual. The ANSI SQL queries where Lingual has “out of the box” support, can be executed using Lingual as-is and others can be implemented using operations from the Cascading API. This way the migration can be rapid and cost effective.
- Use the 80-20 rule: Migrate 80% of the workload using Lingual and Cascading, which does not demand remodelling and rebuilding of ETL, then focus on the remaining 20% of the workload that demand ETL restructuring and rebuilding, using the powerful framework of Cascading.
