Tuesday, December 17, 2013

Apache Kafka - A Different Kind of Messaging System

Apache has released Kafka 0.8, the first major release of Kafka since becoming an Apache Software Foundation top level project. Apache Kafka is publish-subscribe messaging implemented as a distributed commit log, suitable for both offline and online message consumption. It is a messaging system initially developed at LinkedIn for collecting and delivering high volumes of event and log data with low latency. The latest release includes intra-cluster replication and multiple data directories support. Request processing is also now asynchronous, implemented via a secondary pool of request handling threads. Log files can be rotated by age, and log levels can be set dynamically via JMX. A performance test tool has been added, to help fix existing performance concerns and look for potential performance enhancements.

Kafka is a distributed, partitioned, replicated commit log service. Producers publish messages to Kafka topics, and consumers subscribe to these topics and consume the messages. A server in a Kafka cluster is called a broker. For each topic, the Kafka cluster maintains a partition for scaling, parallelism and fault-tolerance. Each partition is an ordered, immutable sequence of messages that is continually appended to a commit log. The messages in the partitions are each assigned a sequential id number called the offset.

Read more here

Leave a Reply

All Tech News IN © 2011 DheTemplate.com & Main Blogger .