Agentic AI Evaluation on Bedrock: Lessons from Pushpay's Journey

Hello everyone,

We're excited to invite you to the Auckland AWS Tools and Programming meetup!

This month, we are diving deep into the realities of moving AI from a cool demo to a production-ready system. We are thrilled to host two fantastic co-speakers: Roger Wang (Senior Solutions Architect @ AWS) and Saurabh Gupta (Pushpay).

We are gathering once again at the Pushpay office in the CBD—looking forward to seeing you all there!

🚨 Action Required: To help us streamline building security and sort out the catering (including dietary requirements), please make sure to pre-register before arriving: 👉 Pre-register for Venue Check-In Here

⚠️ Important Day Change: Please note that we are meeting on a Tuesday this month instead of our usual Thursday slot!

🎤 The Lineup
Roger Wang (AWS) & Saurabh Gupta (Pushpay)

• Title: Breaking Through the 65% Ceiling: How Pushpay Engineered a 95% Accurate Production AI Agent
• Description: Agentic AI plateaus fast once you move beyond demos. Pushpay's natural language search agent hit a ~65% accuracy ceiling before the team built a structured evaluation framework to break through it.
  Roger and Saurabh will walk through Pushpay's real-world journey: building the agent on Amazon Bedrock, why accuracy stalled, and the engineering techniques that pushed it to 95%. They will share practical takeaways including constructing a 300+ query golden dataset, automating measurement with LLM-as-a-Judge, and applying domain-specific prompt engineering on Bedrock. No hype—just real numbers, a real production system, and actionable insights for anyone building or operating agents.

📍 Venue & Day Reminder

• Venue: We are back at the Pushpay office in the Auckland CBD again this month! You can find the address details below.
• Day Change: Don't forget, we are meeting on Tuesday this time around.

📅 Event Details

• When: 5:00 pm on Tuesday, 23rd June 2026 (Note the day change!)
• Where: Pushpay Office, Level 19, 48 Shortland Street, Auckland

⏰ Agenda

• 5:00 pm - Welcome & Networking
• 5:30 pm - Presentation & Deep Dive
• 6:30 pm - Q&A, Announcements, & Networking

This event is proudly supported by Pushpay, Mantel Group, and AWS.
We look forward to seeing you there!
– The Auckland AWS Tools and Programming Team

Code of Conduct
Auckland AWS Tools and Programming is committed to providing a friendly and welcoming space for everyone who attends our meetup events, regardless of gender, gender identity and expression, age, sexual orientation, disability, physical appearance, body size, race, ethnicity, religion (or lack thereof), or technology choices. Any form of harassment, racism, misogyny, or xenophobia is unacceptable. Participants violating these rules will be invited to leave the meetup.