Advanced Automated Testing Techniques (two talks)

*PLEASE NOTE: To confirm your seat to this meetup event, please also RSVP VIA EVENTBRITE (https://www.eventbrite.com/e/advanced-automated-testing-techniques-tickets-16473183736) (this is needed to get past the security).

For our April event, we are really fortunate to have not one, but two awesome speakers who offered to share with us their approaches to automated testing.

Auto-detecting Functional/Performance Regression (50 minutes)

At the end of a refactor, every developer wonders if their code will blow up in production. Unit and integration tests offer some confidence but they typically only cover a small fraction of all possible code paths.

In this talk, Puneet Khanduri from Twitter, offered to introduce us to Diffy, a regression testing (http://en.wikipedia.org/wiki/Regression_testing) framework he created that automatically detects any changes in the functional and performance behaviors of service code.

In his talk, Puneet will cover: what motivated the need for Diffy, how to detect functional changes in behavior of a service, how to detect changes in performance characteristics, how teams use Diffy at Twitter, and show us a live demo.

Automated Test Generation Using Concolic Testing (50 minutes)

In this talk, Koushik Sen from University of California, Berkeley, offered to tell us about the recent advances and challenges in concolic testing (http://en.wikipedia.org/wiki/Concolic_testing) and symbolic execution. Concolic testing, also known as directed automated random testing (DART) or dynamic symbolic execution, is an efficient way to automatically and systematically generate test inputs for programs. Concolic testing uses a combination of runtime symbolic execution and automated theorem proving techniques to automatically generate non-redundant and exhaustive test inputs. Specifically, concolic testing performs symbolic execution along a concrete execution path, generates a logical formula denoting a constraint on the input values, and solves a constraint to generate new test inputs that would execute the program along previously unexplored paths.

A key goal of concolic testing in the context of software testing is to explore as many different program paths as possible in a given amount of time, and for each path to (1) generate a set of concrete input values exercising that path, and (2) check for the presence of various kinds of errors including assertion violations, uncaught exceptions, security vulnerabilities, and memory corruption. The ability to generate concrete test inputs is one of the major strengths of concolic testing: from a test generation perspective, it allows the creation of high-coverage test suites, while from a bug-finding perspective, it provides developers with a concrete input that triggers the bug, which can be used to confirm and debug the error independently of the symbolic execution tool that generated it.

Concolic testing has inspired the development of several industrial and academic automated testing and security tools such as PEX, SAGE, and YOGI at Microsoft, Apollo at IBM, Conbol at Samsung, and CUTE, jCUTE, CATG, Jalangi, SPLAT, BitBlaze, jFuzz, Oasis, and SmartFuzz in academia. A central reason behind the wide adoption of concolic testing is that, while concolic testing uses program analysis and automated theorem proving techniques internally, it exposes a testing usage model that is familiar to most software developers.

About Puneet Khanduri
Puneet is a member of the newly formed Test Frameworks team within the Engineering Effectiveness. His team focuses on addressing technology gaps in Twitter's testing capabilities. Prior to Twitter, Puneet worked at Oracle Labs where he built a real-time analytics platform for live sensor data. Prior to that, Puneet worked as a researcher at Sun Labs and benchmarked network and microprocessor architectures using trace-driven simulations. Follow him at @pzdk (https://twitter.com/pzdk).

About Koushik Sen
Koushik Sen is an associate professor in the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley. His research interest lies in Software Engineering, Programming Languages, and Formal methods. He is interested in developing software tools and methodologies that improve programmer productivity and software quality. He is best known for his work on “DART: Directed Automated Random Testing” and concolic testing. He has received a NSF CAREER Award in 2008, a Haifa Verification Conference (HVC) Award in 2009, a IFIP TC2 Manfred Paul Award for Excellence in Software: Theory and Practice in 2010, a Sloan Foundation Fellowship in 2011, and a Professor R. Narasimhan Lecture Award in 2014. He has won several ACM SIGSOFT Distinguished Paper Awards. He received the C.L. and Jane W-S. Liu Award in 2004, the C. W. Gear Outstanding Graduate Award in 2005, and the David J. Kuck Outstanding Ph.D. Thesis Award in 2007, and a Distinguished Alumni Educator Award in 2014 from the UIUC Department of Computer Science. He holds a B.Tech from Indian Institute of Technology, Kanpur, and M.S. and Ph.D. in CS from University of Illinois at Urbana-Champaign.