From Gut Feelings to Data: Quantifying Proctor Judgment in High-Stakes Exams

As high-stakes exams move online, ensuring their integrity poses both technical and human challenges. This talk focuses on quantifying the quality of human judgment in proctoring—a critical but often overlooked aspect of test security. Like other professionals, proctors make high-stakes decisions under uncertainty, with judgments influenced by lapses in attention, inconsistent training, and individual biases. We introduce the Professional Calibration Tool (PCT), a psychometric system that both measures and increases decision accuracy and reliability by embedding stealth assessments—pre-adjudicated cases—into live workflows and delivering real-time feedback based on expert and peer consensus.