Maintaining Consistency:Annotation Guideline Management for Digital & Voice Bots

Consistency is key when it comes to labeling data. At Posh, our digital and voice bots in the financial domain consist of many context-dependent turns of dialogue. The complexity of our data requires human annotators to follow a set of clear guidelines to help ensure consistency and high inter-annotator agreement. In this talk, I'll walk you through the steps of writing guidelines for an annotation task.

• Overview: Posh's Prodigy cloud annotation service
• Writing Guidelines: Where to Begin?
• Maintenance and Quality-Control: Dealing with Subjective and Contextual Content
• Looking to the Future: Generating Examples and Labeling Data using GPT-3

Agenda:
7:00 - Welcome
7:05 - Maintaining Consistency:Annotation Guideline Management for Digital & Voice Bots by Cheyanne Baird + Q&A
7:55 - Wrap-up