Tuesday, November 26, 2013

First steps to building a scalable high volume messaging system

At Ancestry.com we are becoming more data driven. That means we want to capture more data about our systems, including how our users are interacting with them. Part of that strategy is to capture the log files from our application servers and put them into our Hadoop cluster.

We have tried using MSMQ and RabbitMQ in a custom framework to capture messages, but found that it does not scale enough for our requirements. So we are trying out a new messaging framework from LinkedIn called, Kafka .

Kafka is a distributed open source messaging system that captures logs into files. It is designed to handle very large volumes of data in the order of 10-50 MB/sec, depending on the configuration of the Kafka cluster.

Read more here

Leave a Reply

All Tech News IN © 2011 DheTemplate.com & Main Blogger .