Embrace:AI // 2025.02: Evaluating LLMs + Red Teaming GenAI Applications

Agenda:

• 18h: Welcome and networking
• 18h20: Intro - Devoteam (Nuno Cancelo)
• 18h30: Evaluating LLMs: From RAG Pipelines to Advanced Reasoning (Abi Aryan)
• 19h00: LLMs Under Siege: Red Teaming GenAI Applications (João Galego)
• 19h30-21h30: Networking with Free Drinks & Pizza (Sponsored by Devoteam)

Join and help build the AI community in Lisbon!
RSVP now and invite your peers.

*** The Lineup for the evening
Talk 1: Evaluating LLMs: From RAG Pipelines to Advanced Reasoning
As language models become the leading OS for AI applications, robust evaluation methods becomes the key to ensuring their reliability. This talk will explore evaluation strategies for both Retrieval-Augmented Generation (RAG) systems and reasoning-focused models. We will discuss key metrics, and benchmarking approaches tailored to different use cases. Attendees will gain insights into assessing retrieval effectiveness, factual consistency, and reasoning capabilities, along with practical techniques to improve model performance. Whether you're building production-ready LLM applications or researching model behaviors, this session will provide actionable frameworks for systematic evaluation.

Speaker: Abi is the founder of Abide AI and the author of OReilly book LLMOps: Managing LLMs in Production. With a background in research and MLOps, Abi is passionate about building robust, scalable AI systems. With experience spanning AutoML, multimodal models and multiagent systems, they actively share insights through speaking engagements, workshops, and research. She is also currently writing her next book on GPU engineering for AI systems with Packt Publications.

Social Media:
www.linkedin.com/in/goabiaryan/
www.x.com/GoAbiAryan

***

Talk 2: LLMs Under Siege: Red Teaming GenAI Applications

Speaker: João Galego is Principal AI Engineer / Head of AIOps at Critical Software and Invited Professor at ISEG. He has a MSc in Physics from IST and a PgD in Forensic Studies from ISCSP. Before Critical, he worked as a Lead ML Engineer at Siemens and as a Solutions Architect, ML Specialist, and Quantum ambassador within the Startups team at AWS. When he's not working or teaching, you can usually find him browsing the local bookstore, contributing to open source or enjoying some quality family time.

Social Media: www.linkedin.com/in/jgalego/

*** About our Sponsor

Big shoutout and thank you to our sponsor, Devoteam, for providing the venue, drinks, and pizza!
Devoteam is an AI-driven tech consulting firm specialised in cloud platforms, cyber, data, and sustainability.
Tech native for almost 30 years, Devoteam guides businesses through sustainable digital transformation to deliver value.
With over 11,000 tech architects in more than 25 countries across Europe, the Middle East, and Africa, Devoteam is committed to using technology to serve people.

www.devoteam.com
Nuno Cancelo: www.linkedin.com/in/nunocancelo/

*** About Embrace:AI

Join Embrace:AI, be part of an open meetup fostering a vibrant AI community that brings together engineers, researchers, data scientists, and leaders to explore cutting-edge advancements, real-world use cases, and critical topics like Responsible AI, security, and governance.

You'll have access to high-quality, non-commercial content that covers both the technical and non-technical aspects of AI. Concise, impactful presentations ensure you gain valuable insights while still having plenty of time for in-person networking.

Experience the power of face-to-face connection—meet like-minded professionals and shape the future of this transformative technology. Together, we'll embrace the limitless potential of AI.
We are looking forward to seeing you at the event!
Join and help build the AI community in Lisbon!

RSVP now and invite your peers.