FraudSMSWalker: Benchmarking Agentic Large Language Models for SMS-to-Webpage Fraud Detection

2026-06-15 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

FraudSMSWalker is a new benchmark designed to evaluate agentic Large Language Models (LLMs) for cross-channel SMS-to-webpage fraud detection. This benchmark addresses limitations in existing evaluations that either focus solely on message content or allow models to exploit URL and domain reputation shortcuts. FraudSMSWalker comprises 699 bilingual chains, including 332 fraudulent and 367 benign cases, spanning ten distinct service scenarios. Crucially, the benchmark masks raw URLs, hosts, domains, IPs, redirects, and reputation metadata, providing models only with SMS context and sanitized webpage evidence. It also features "hard benign" cases with login or payment elements that are legitimate within context but mimic scam flows. Initial evaluations of nine web agents reveal they can identify suspicious cues but struggle with benign recall and often generate weakly supported positive predictions. This positions FraudSMSWalker as a critical tool for assessing evidence-grounded fraud judgments when direct reputation shortcuts are unavailable.

Key takeaway

For AI Security Engineers developing fraud detection systems, this research highlights a critical vulnerability: current agentic LLMs often fail to make accurate, evidence-grounded fraud judgments without URL reputation data. You should prioritize training models on content alignment between SMS and webpages, especially for "hard benign" cases. Focus on improving benign recall to avoid false positives, ensuring your systems can detect sophisticated cross-channel scams even when direct URL cues are suppressed.

Key insights

Agentic LLMs struggle with evidence-grounded SMS-to-webpage fraud detection when URL reputation shortcuts are removed.

Principles

• Cross-channel fraud requires SMS-to-webpage alignment.
• Masking URLs prevents reputation shortcut reliance.
• Hard benign cases test nuanced judgment.

Method

FraudSMSWalker evaluates web agents using URL-masked browser protocols, providing SMS context and sanitized webpage evidence for fraud judgment across ten service scenarios.

In practice

• Focus LLM training on content alignment.
• Develop agents for masked URL environments.
• Prioritize benign recall in fraud models.

Topics

• SMS Fraud Detection
• Agentic Large Language Models
• Cross-channel Fraud
• Webpage Analysis
• Benchmark Evaluation
• URL Masking

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• How Much Can We Trust LLM Search Agents? Measuring Endorsement Vulnerability to Web Content Manipulation — LLM search agents exhibit varied endorsement vulnerability to web content manipulation, requiring robust safety evaluations.
• AI Is Turbocharging the Spamosphere, Amping Up Prolific Text-Message Scams — AI, particularly Google's Gemini model, is being exploited to create sophisticated fake websites, turbocharging text-message scams and financial losses.
• A Distribution-Free Framework for Rewrite-Based Human-text Detection via Knockoff Filtering — A distribution-free framework provides finite-sample FDR guarantees for rewrite-based human-text detectors without retraining.
• Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce — The two-agent simulation framework effectively evaluates agentic search architectures and reveals key performance and evaluation challenges.
• What this year’s Black Friday taught security teams about agentic commerce — Agentic commerce blurs lines between legitimate and malicious automated traffic, requiring new security paradigms.
• Mozilla says 271 vulnerabilities found by Mythos have "almost no false positives" — An AI agent harness significantly reduces false positives in vulnerability detection by integrating LLMs into existing developer toolchains.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.