Tuesday, December 9, 2014

BUILDING AN INEXPENSIVE PETABYTE DATABASE WITH MONGODB AND AMAZON WEB SERVICES

0 comments
I’ve always been fascinated by questions of scalability. Working for then-leading enterprise search vendor Verity around 2005, we developed a sub-specialty implementing text search applications for our largest customers. Anything over 30 million documents stretched the already obsolescent Verity engine to its limits. Such implementations were clearly a level beyond mere “enterprise scale,” and there was no common term for larger problems, so we started referring to them as “empire scale.”

Today, standard terminology in IT has converged on the term “Big Data,” to describe problems whose very scale dictates the software and techniques used to process them. It also describes a good part of what we at MongoDB address with our customers. Thousands of organizations have adopted MongoDB, primarily because it has enabled fast deployment and agility at all dimensions of scalability, such as databases providing subsecond response over billions of documents, or hosting hundreds of thousands of mobile applications on a single platform.

Challenge

With customers having explored so many dimensions of scalability, we wanted to explore scalability in terms of a data size that would be instantly recognizable, but in practical terms. We settled on a petabyte of data, with the goal of creating a database as inexpensively as possible.

How much is a petabyte? Under the SI system, it’s abbreviated PB and equal to 10^15 bytes, not to be confused with IEC system’s “pebibyte”, abbreviated as PiB and equal to 2^50. Since petabyte is the more common standard, we chose that as our target.

Read here Part 1

Leave a Reply

 
All Tech News IN © 2011 DheTemplate.com & Main Blogger .