MoT Norwich #2 - Walking the Talk: Random Exploration of a Chatbot API

Well... let's try that again :D

We're joined by James Thomas, Senior Expert Quality Engineer at ADA health and Vice President of Association for Software Testing. James will be sharing his experiences on the challenges of testing chatbots and the approach he took. Here's what James' has shared about his upcoming talk:

Three related coverage risks stood out when I begain to test a chatbot API for a medical symptom checker. With an infinite space of possible chats, how could we:

• look for unintended consequences of changes as we built the API
• discover some of the edge and corner cases bugs that would surely exist
• exercise the API to any significant extent after each iteration

To help mitigate these risks I built a client which would randomly walk through dialogs, unattended, and report on what it had found.

In this talk, I'll describe how I implemented that client by iteratively adding functionality that I hoped would facilitate my exploration of changes and fixes to the emerging API. I'll give examples of features that worked well (such as configuration of probabilities for different types of answers) and those that did not (such as checking for specific classes of medical outcome), explain how I built on top of the client to make a load testing tool, and think about what I'd do differently next time.