Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in LLMs
Details
Zhiyang Chen (University of Toronto & ETH Zurich) will give a seminar on 04 Sep 2026 at 15:00 in KTH Room 1440 (Henrik Eriksson) and on Zoom.
Title
Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs
Abstract
Large language models are increasingly used for software development, but their generated code can sometimes incorporate malicious URLs and endpoints originating from scam websites, even in response to seemingly benign developer requests. In this talk, I will present Scam2Prompt, a scalable framework for auditing this behavior in production LLMs. Scam2Prompt starts from known malicious websites and automatically constructs realistic developer-style prompts to test whether LLMs reproduce scam infrastructure in generated code. Using this framework, we build Innoc2Scam-bench, a renewable benchmark of 1,377 prompts, and evaluate several state-of-the-art production LLMs. Our results show that malicious endpoint generation remains prevalent across recent models. Interestingly, the framework also uncovered previously unknown active scam websites, 62 of which were subsequently confirmed and added to MetaMask’s scam database. More broadly, this work highlights a security risk beyond explicit harmful prompting: LLMs may unintentionally propagate unsafe web content when assisting users with executable code.
Bio
Zhiyang Chen is currently visiting ETH Zurich until Feb 2027, working with Prof. Zhendong Su, He is a final-year PhD candidate at the University of Toronto advised by Prof. Fan Long. His research focuses on blockchain and smart contract security, and more recently, AI security and efficient agentic systems. More broadly, he has also collaborated on research involving compilers and program analysis, as well as CUDA kernel generation. Zhiyang is also an active contributor to open-source software such as SQLite. More information about his work can be found at https://jeffchen006.github.io/.
You can also join by zoom at https://kth-se.zoom.us/j/64568739111