Characterizing Tests in IoT Software: Practices, Challenges and Opportunities

2026-06-12 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Software Development & Engineering, Internet of Things (IoT) & Connected Devices · Depth: Expert, extended

Summary

An empirical study characterized testing practices in open-source Internet of Things (IoT) software, analyzing 824 projects and conducting in-depth coverage and mutation analysis on 37 runnable repositories. The research found that while IoT software often includes a considerable number of tests, their overall effectiveness remains limited, with average statement coverage of 65.2%, branch coverage of 53.4%, and a mutation score of 39.9%. A key finding is that over half (51.4%) of uncovered code segments stem from challenges in managing complex interactions with external dependencies, particularly network communication scenarios. The study also revealed that 76.04% of mock objects in IoT tests simulate network APIs, aligning directly with these identified testing hurdles and suggesting mocking as a crucial solution for enhancing test coverage.

Key takeaway

For software engineers developing IoT systems, you should prioritize robust mocking strategies to address the pervasive challenges of external dependencies, especially network communications. Your test suites must effectively simulate diverse protocols and device interactions. Consider exploring mock object migration from well-tested open-source projects to enhance coverage and reduce manual effort in creating context-specific test doubles.

Key insights

IoT software testing effectiveness is limited by complex external dependencies, especially network communications, making mock objects crucial.

Principles

• IoT testing faces unique challenges beyond traditional software.
• External dependencies are primary barriers to IoT test coverage.
• Mock objects are essential for simulating IoT communication.

Method

The study conducted static analysis on 824 IoT projects, then dynamic coverage and mutation analysis on 37 runnable projects. It used open coding for uncovered code and closed coding for mock object roles and simulated dependencies.

In practice

• Use mock objects to simulate network communication APIs.
• Consider migrating existing mocks from open-source IoT projects.
• Focus on protocol-aware mocking for complex dependencies.

Topics

• IoT Software Testing
• Test Coverage Analysis
• Mock Object Usage
• External Dependencies
• Network Communication Protocols
• Open-Source Software

Best for: AI Scientist, Research Scientist, Software Engineer

Related on AIssential

• Pytest Tutorial: MLOps Testing, Fixtures, and Locust Load Testing — Robust MLOps requires a layered testing strategy encompassing unit, integration, and performance tests for reliability.
• Article: Building a Secure MCP Server on AWS for a Million-Company B2B Platform — Treating MCP servers as first-class interfaces with narrow tool contracts enhances security and maintainability for LLM integrations.
• Integrating Business Needs and Technical Skills in Effective Model Serving Deployments - ML 184 — Effective model serving requires integrating business needs with technical skills through iterative design and robust testing.
• How a Small Enablement Team Supported Adopting a Single Environment for Distributed Testing — A small enablement team and proxy routing enabled scalable distributed system testing within a single development environment.
• BugBash'26 Morning of Day 1 — Peter Alvaro's 20-year journey in distributed systems reveals no silver bullet, but continuous progress through diverse, evolving approaches.
• Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation — KTester enhances LLM-based unit test generation by integrating project-specific and testing domain knowledge through a modular, multi-perspective pipeline.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.