Friday, August 1, 2014

Reliable Queueing in Redis

Here in Bronto Engineering, we sure do love us some Redis. And why not? It is fast, flexible, and has a rock solid track record in our production environment. It isn’t surprising that when we were looking for a better implementation for our distributed work queues, we decided to take a look at what Redis could provide.

The Need For Reliable Queueing

The asynchronous, distributed processing of work queues is a common pattern within our services. Most of the major features of our platform (e.g. automation workflows, real-time segmentation) rely on these queues, which help us to dynamically scale our processing capacity, and act as a protective buffer during times of high demand.

Reliable queues add an additional guarantee: once an item has been added to the queue, it will not be permanently removed until that item has been confirmed as being fully processed. In most implementations, this means that the consumer is going to interact with the queue at least twice, once to pop a piece of work from the queue, and again to mark that piece of work as complete.

Read more here

Leave a Reply

All Tech News IN © 2011 & Main Blogger .