Wednesday, February 4, 2015

Improving on the Lambda Architecture for streaming analysis

Modern organizations have started pushing their big data initiatives beyond historical analysis. Fast data creates big data, and applications are being developed that capture value, specifically real-time analytics, the moment fast data arrives. The need for real-time analysis of streaming data for real-time analytics, alerting, customer engagement or other on-the-spot decision-making, is converging on a layered software setup called the Lambda Architecture.

The Lambda Architecture, a collection of both big and fast data software components, is a software paradigm designed to capture value, specifically analytics, from not only historical data, but also from data that is streaming into the system.

In this article, I’ll explain the challenges that this architecture currently presents and explore some of the weaknesses. I’ll also discuss an alternative architecture using an in-memory database that can simplify and extend the capabilities of Lambda. Some of the enhancements to the Lambda Architecture that will be discussed are:

  • The ability to return real-time, low-latency responses back to the originating system for immediate actions, such as customer-tailored responses. Data doesn’t have to only flow one way, into the system. 
  • The addition of a transactional, consistent (ACID) data store. Data entering the system can be transactional and operated upon in a consistent manner. 
  • The addition of SQL support to the speed layer, providing support for ad hoc analytics as well as support for standard SQL report tooling.
read more here

Leave a Reply

All Tech News IN © 2011 & Main Blogger .