How to Compare NoSQL Databases: Determining True Performance and Recoverability

Hosted by Scale Warriors of NYC

Public group

This is a past event

50 people went


3 Park Avenue, 27th Floor · New York, NY

How to find us

Two blocks from the Empire State Building between 33rd St and 34th St and Park Ave.

Location image of event venue


One of the major challenges in choosing an appropriate NoSQL solution is finding reliable information as to how a particular database performs for a given use case. Stories of high profile systems failures abound, and tossed around with widely varying benchmark numbers that seem to have no bearing on how tuned systems behave out of the lab. Many of the profiling tools and studies out there use deeply flawed tools or methodologies. Getting meaningful data out of published benchmark studies is difficult, and running internal benchmarks even more so.

In this presentation, Will Keung, Senior Software Engineer at adMarketplace, will talk about choosing the best NoSQL engine and avoiding vendor lock with a smart NoSQL abstraction layer.

Ben Engber, CEO and founder of Thumbtack Technology, will discuss how to perform tuned benchmarking across a number of NoSQL solutions (Couchbase, Aerospike, MongoDB, Cassandra, HBase, others) and to do so in a way that does not artificially distort the data in favor of a particular database or storage paradigm. This includes hardware and software configurations, as well as ways of measuring to ensure repeatable results.

We also discuss how to extend benchmarking tests to simulate different kinds of failure scenarios to help evaluate the maintainablility and recoverability of different systems. This requires carefully constructed tests and significant knowledge of the underlying databases — the talk will help evaluators overcome the common pitfalls and time sinks involved in trying to measure this.

Lastly we discuss the YCSB benchmarking tool, its significant limitations, and the significant extensions and supplementary tools Thumbtack has created to provide distributed load generation and failure simulation.


WILL KEUNG - senior software engineer, adMarketplace

William has been a software engineer for more than ten years, during which time he has designed and implemented enterprise frameworks for various Fortune 500 companies. He has worked in different industries such as telecomm, banking and finance, publishing, biotech and the US government. His specialty is Java with a hint of security and networking. In his spare time, he tracks the startup and the local culinary scene while dabbling on Ruby on Rails and natural language processing.

BEN ENGBER - CEO and Founder of Thumbtack Technology

Ben has made a career around building highly reliable internet-scale applications. Prior to founding Thumbtack Technology in 2003 he was VP of Operations Technology at, where he built the back-office and e-commerce architectures needed to support millions of digital and physical transactions per day. During his tenure he became acutely aware of the need for outsourced labor that could both understand core business concerns and drive high-quality and robust software infrastructure projects. His inability to find such a partner led him to establish Thumbtack.

Ben holds a BS in Mathematics from the University of Chicago and a MBA from Boston University.