Name: August 7 - Understanding Visual Agents
Start: 2025-08-07T12:00:00-04:00
End: 2025-08-07T14:00:00-04:00

Join us for a virtual event to hear talks from experts on the current state of visual agents.

**When**

Aug 7, 2025 at 9 AM Pacific

**Where**

Virtual. **[Register for the Zoom.](https://voxel51.com/events/understanding-visual-agents-august-7-2025)**

**Foundational capabilities and models for generalist agents for computers**

As we move toward a future where language agents can operate software, browse the web, and automate tasks across digital environments, a pressing challenge emerges: how do we build foundational models that can act as generalist agents for computers? In this talk, we explore the design of such agents—ones that combine vision, language, and action to understand complex interfaces and carry out user-intent accurately.

We present OmniACT as a case study, a benchmark that grounds this vision by pairing natural language prompts with UI screenshots and executable scripts for both desktop and web environments. Through OmniACT, we examine the performance of today’s top language and multimodal models, highlight the limitations in current agent behavior, and discuss research directions needed to close the gap toward truly capable, general-purpose digital agents.

*About the Speaker*

[Raghav Kapoor](https://www.linkedin.com/in/raghavkapoor16/) is a machine learning at Adobe, where he works on the Brand Services team, contributing to cutting-edge projects in brand intelligence. His work blends research with machine learning, reflecting his deep expertise in both areas. Prior to joining Adobe, Raghav earned his Master’s degree from Carnegie Mellon University, where his research focused on multimodal machine learning and web-based agents. He also brings industry experience from his experience as a strategist at Goldman Sachs India.

**BEARCUBS: Evaluating Web Agents' Real-World Information-Seeking Abilities**

The talk focuses on the challenges of evaluating AI agents in dynamic web settings, the design and implementation of the BEARCUBS benchmark, and insights gained from human and agent performance comparisons. In the talk, we will discuss the significant performance gap between human users and current state-of-the-art agents, highlighting areas for future improvement in AI web navigation and information retrieval capabilities.

*About the Speaker*

[Yixiao Song](https://www.linkedin.com/in/songyixiao/) is a Ph.D. candidate in Computer Science at the University of Massachusetts Amherst. Her research focuses on enhancing the evaluation of natural language processing systems, particularly in assessing factuality and reliability in AI-generated content. Her work encompasses the development of tools and benchmarks such as VeriScore, an automatic metric for evaluating the factuality of long-form text generation, and BEARCUBS, a benchmark for assessing AI agents' ability to identify factual information from web content.

**Visual Agents: What it takes to build an agent that can navigate GUIs like humans**

We’ll examine conceptual frameworks, potential applications, and future directions of technologies that can “see” and “act” with increasing independence. The discussion will touch on both current limitations and promising horizons in this evolving field.

*About the Speaker*

[Harpreet Sahota](https://www.linkedin.com/in/harpreetsahota204/) is a hacker-in-residence and machine learning engineer with a passion for deep learning and generative AI. He’s got a deep interest in RAG, Agents, and Multimodal AI.

**Implementing a Practical Vision-Based Android AI Agent**

In this talk I will share with you practical details of designing and implementing Android AI agents, using [deki.](http://github.com/RasulOs/deki)

From theory we will move to practice and the usage of these agents in
industry/production.

For end users - remote usage of Android phones or for automation of standard tasks. Such as:

* "Write my friend 'some_name' in WhatsApp that I'll be 15 minutes late"
* "Open Twitter in the browser and write a post about 'something'"
* "Read my latest notifications and say if there are any important ones"
* "Write a linkedin post about 'something'"

And for professionals - to enable agentic testing, a new type of test that only became possible because of the popularization of LLMs and AI agents that use them as a reasoning core.

*About the Speaker*

[Rasul Osmanbayli](https://www.linkedin.com/in/rasulosmanbayli/) is a senior Android developer at Kapital Bank, Baku/Azerbaijan. It is the largest private bank in Azerbaijan. He created deki, an Image Description model that was used as a foundation for an Android AI agent that achieved high results on 2 different benchmarks: Android World and Android Control.

He previously worked in Istanbul/Türkiye for various companies as an
Android and Backend developer. He is also a MS at Istanbul Aydin University in Istanbul/Türkiye.

Jimmy Guerrero

New York AI, Machine Learning and Computer Vision Meetup

AI, Machine Learning and Computer Vision Meetup Network

Technology

Open Source

Machine Learning

Data Science

AI/ML

Artificial Intelligence

Computer Vision

August 7 - Understanding Visual Agents

Online event

Share

New York AI, Machine Learning and Computer Vision Meetup

August 7 - Understanding Visual Agents

New York AI, Machine Learning and Computer Vision Meetup

Details

Members are also interested in