Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Internet of Things (IoT) & Connected Devices · Depth: Expert, long

Summary

An empirical benchmark evaluated five large language models—ChatGPT-4o, Gemini 2.5 Flash, DeepSeek, Kimi, and Llama 3.1 8B—on their ability to assess multi-sensor physical hazard data. Across 60 scenarios and 1,800 API calls, the study found that all tested models consistently failed to produce precautionary warning signals when multiple sensors were simultaneously elevated below individual safety limits, even when combined exposure indicated a hazard (e.g., OSHA additive exposure index $E_m > 1.0$). Conversely, these models achieved near-perfect accuracy (0.975–1.000) on single-sensor threshold violations. Structured tabular input formatting offered no consistent advantage and significantly reduced ChatGPT-4o's performance ($p=0.001$). Action recommendation quality varied widely, with DeepSeek and Gemini 2.5 Flash performing best.

Key takeaway

For practitioners deploying large language models in physical safety monitoring systems, you must explicitly validate their performance on multi-sensor joint assessment scenarios. Your systems relying on these models without specific joint assessment prompting risk producing false-safe outputs in combined-elevation conditions. Consider implementing explicit chain-of-thought prompting, additive index computation, or rule-based post-processing to address this critical capability gap.

Key insights

LLMs reliably detect single-sensor violations but fail to identify hazards from combined, sub-threshold multi-sensor data.

Principles

• Multi-sensor joint assessment is a critical LLM capability gap.
• Single-sensor threshold arithmetic is a reliable LLM capability.
• Input format impact varies by model, not universally beneficial.

Method

The benchmark involved 1,800 API calls to five LLMs across 60 scenarios, evaluating multi-sensor joint assessment, response proportionality, and pattern disambiguation using a three-question scoring rubric ($Q_1, Q_2, Q_3$) at temperature 0.0.

In practice

• Validate LLMs on multi-sensor joint scenarios.
• Consider explicit chain-of-thought prompting.
• Implement rule-based post-processing for combined hazards.

Topics

• Large Language Models
• Multi-sensor Systems
• Physical Hazard Assessment
• LLM Benchmarking
• Safety Monitoring Systems
• Prompt Formatting

Code references

• Faizaniqbal52/PhysicalHazardBenchmark

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most — More capable LLMs make worse forecasts for superlinear growth and tail risk, especially in the upper tail.
• Benchmarking the Safety of Large Language Models for Robotic Health Attendant Control — Current LLMs exhibit high safety violation rates as robotic health attendant controllers, precluding clinical deployment.
• Zero-Shot Goal Recognition with Large Language Models — LLM goal recognition competence varies significantly, with some models integrating evidence effectively while others rely on world-knowledge priors.
• How catastrophic is your LLM? — C3LLM statistically certifies LLM catastrophic risks in multiturn conversations, moving beyond empirical spot-checking.
• Safety and accuracy follow different scaling laws in clinical large language models — Clinical LLM safety is a deployment property, not a passive consequence of scaling model size or compute.
• From Rubrics to Recipe: Principle-Centric Benchmark for Evaluating Large Language Models — Evaluating LLMs with human-readable principles improves output quality and scales assessment beyond manual curation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.