Independent Test Generation for RESTful APIs

2024-01-31 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Advanced, extended

Summary

BosqTGen is a novel black-box methodology and tool for RESTful API test generation, introduced in 2025. It addresses challenges like polyglot systems, source code inaccessibility, and structured input generation by decomposing API specifications into primitives. The system uses LLMs to suggest coherent strata for these primitives and employs combinatorial testing to efficiently sample values, avoiding random sampling redundancy. BosqTGen achieves an average of 82% code coverage on 5 standard RESTful benchmarks, representing a 20% or more increase over prior systems like RESTler, ARAT-RL, EvoMaster, and AutoRestTest. It nears parity with hand-written test suites, with effective branch coverage rising to 100% for "Person Controller" and 93% for "User Management" after accounting for unreachable auto-generated code. The tool also incorporates mock-aware test generation, boosting coverage by over 20% in specific cases.

Key takeaway

For software engineers or QA professionals building and testing RESTful APIs, BosqTGen offers a significant advancement in automated test generation. You should consider integrating this black-box methodology to achieve higher code coverage, potentially reducing the need for extensive manual test suite creation. Its LLM-driven, combinatorial approach, especially with mock-aware generation, can uncover critical interactions and improve test reliability for complex, polyglot systems.

Key insights

BosqTGen combines LLMs and combinatorial testing to generate high-coverage, black-box RESTful API tests.

Principles

• Decompose complex API inputs into primitive types for easier partitioning.
• Leverage LLMs with local and global context for representative value generation.
• Apply combinatorial testing's interaction effect hypothesis to finite dynamic input domains.

Method

BosqTGen decomposes API specifications into primitive components, uses LLMs with contextual prompts to generate stratified values, then applies combinatorial testing to efficiently combine these values into structured inputs, incorporating mock data for enhanced coverage.

In practice

• Use API specification languages like BosqCon, OpenAPI, or Spring.
• Provide LLMs with API signature, field paths, and mock data.
• Limit collection sizes (e.g., 0-3 elements) to manage combinatorial explosion.

Topics

• RESTful API Testing
• Test Generation
• Combinatorial Testing
• Large Language Models
• Code Coverage
• API Specifications

Code references

• google/AFL
• codingsoo/ARAT-RL

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• LLMCFG-TGen: Using LLM-Generated Control Flow Graphs to Automatically Create Test Cases from Use Cases — Coupling LLM semantic reasoning with structured Control Flow Graph modeling effectively bridges NL requirements to systematic test generation.
• Building an A/B testing analysis framework for mobile gaming on Databricks — A Databricks-native framework automates A/B testing, standardizing analysis and delivering AI/BI-powered insights.
• A new era for software testing — LLMs can significantly enhance software QA and testing by automating complex, traditionally manual processes without compromising quality.
• MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysis — Functional coupling between methods can automate metamorphic relation construction and MTC generation, addressing the test oracle problem.
• Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation — KTester enhances LLM-based unit test generation by integrating project-specific and testing domain knowledge through a modular, multi-perspective pipeline.
• Meta Applies Mutation Testing with LLM to Improve Compliance Coverage — LLMs can significantly improve mutation testing by generating context-aware mutants and targeted tests, enhancing compliance coverage.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.