Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding

2025-05-17 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

This paper introduces and evaluates Retrieval-Augmented Generation (RAG) and Constrained Decoding (CD) as methods to improve the correctness of Large Language Model (LLM)-generated web API invocation code. Researchers designed a RAG retriever for OpenAPI specifications and an automatic regex-based constraint generator for CD. Evaluating 24 open-source LLMs on WAPIIBench's synthetic dataset and a new real-world GitHub-derived dataset, the study found that vanilla LLMs achieved only 13% average correctness for full completion on the synthetic dataset. RAG reduced hallucinations and increased correctness by 113% for full completion on the synthetic dataset, but showed negligible impact elsewhere due to models generating unnecessary parameters. CD consistently eliminated illegal URLs, HTTP methods, and arguments, boosting correctness by 209% and 143% on the synthetic dataset for full and argument completion, respectively. The RAG+CD combination achieved the highest correctness scores (up to 73%) but exhibited less consistency than CD alone.

Key takeaway

For Machine Learning Engineers building LLM-powered code generation tools for web API integrations, prioritize implementing constrained decoding (CD). CD offers consistent and substantial correctness gains, eliminating hallucinations of URLs, HTTP methods, and arguments. While Retrieval-Augmented Generation (RAG) can further boost performance, especially for full completion, its benefits are model-dependent and can lead to parameter overuse. Evaluate RAG's impact carefully for your specific models; otherwise, CD alone provides a more robust and safer path to reliable API invocation code.

Key insights

Constrained decoding reliably eliminates LLM hallucinations in web API invocation code, significantly boosting correctness.

Principles

• OpenAPI specifications can drive RAG context.
• Regex constraints enforce API compliance.
• RAG benefits are model-dependent.

Method

A retriever processes OpenAPI specifications for compact endpoint representations in prompts. An automatic translator converts OpenAPI specifications into regex-based constraints for generation.

In practice

• Use CD to prevent illegal API calls.
• Validate RAG benefits per model.
• Combine RAG+CD for peak correctness.

Topics

• LLM Code Generation
• Web API Integration
• Retrieval-Augmented Generation
• Constrained Decoding
• OpenAPI Specification
• Code Hallucinations

Code references

• stg-tud/WAPIIBench
• manchenkoff/openapi3-parser
• ctimmerm/axios-mock-adapter
• mrabarnett/mrab-regex

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• RAG Explained Through an Exam Analogy — Retrieval-Augmented Generation (RAG) grounds LLM responses in external, current data, reducing hallucinations and overcoming knowledge cutoffs without costly retraining.
• SWE-InfraBench: Evaluating Language Models on Cloud Infrastructure Code — LLMs struggle with imperative IaC code modification, but iterative feedback and RAG significantly improve performance.
• The Open Source RAG Stack: A Complete Guide to Building Retrieval-Augmented Generation Systems — The open-source RAG stack offers a modular, customizable approach to building context-rich AI systems.
• Context-as-a-Service: Surfacing Cross-File Dependency Chains for LLM-Generated Developer Documentation — CaaS enhances LLM documentation agents by surfacing non-obvious cross-file dependencies, improving accuracy and efficiency.
• LLM-Powered Deep Parsing for Industrial Inventory Search — LLM-powered deep parsing extracts structured data from messy industrial text for improved search and deduplication.
• HPC-LLM: Practical Domain Adaptation and Retrieval-Augmented Generation for HPC Support — Domain adaptation and RAG enable smaller LLMs to match larger models in specialized HPC support with fewer resources.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.