Case study: solving P-99 with LPTP and an LLM

2026-07-23 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

An experiment successfully solved the first 33 problems of the Ninety-Nine Prolog Problems (P-99) using Claude from Anthropic, a Large Language Model, in conjunction with LPTP (Logic Program Theorem Prover). The process involved prompting Claude with informal English specifications to generate Prolog code and corresponding test files. These tests were then run, and LPTP was used to formally prove types, groundness, termination, uniqueness, existence, and sometimes functional correctness of the generated code. This "vibe-coding/vericoding" approach resulted in Claude producing 58 logic procedures, 508 tests, and 257 lemmas, totaling 11800 proof lines, all manually checked for accuracy and proof-checked with LPTP.

Key takeaway

For Research Scientists exploring AI-assisted code generation, this case study demonstrates a robust methodology for ensuring reliability. You should integrate formal verification tools like LPTP into your LLM-driven development workflows to validate generated code for critical properties such as termination and correctness. This approach mitigates risks associated with "vibe-coding" by providing concrete proof of code quality.

Key insights

Combining LLM code generation with formal verification significantly enhances code reliability and correctness.

Principles

• LLMs can generate complex, verifiable code.
• Formal provers validate LLM outputs.
• Informal specs can drive code generation.

Method

Prompt an LLM with informal specifications to generate code and tests. Run tests, then use a formal prover like LPTP to verify properties such as types, termination, and functional correctness.

In practice

• Use LLMs for initial code drafts.
• Integrate formal verification tools.
• Apply to logic programming tasks.

Topics

• Prolog
• Large Language Models
• Formal Verification
• Logic Program Theorem Prover
• Code Generation
• AI Reliability

Best for: AI Scientist, Research Scientist, NLP Engineer

Related on AIssential

• The TechBeat: Can LLMs Generate Quality Code? A 40,000-Line Experiment (1/6/2026) — LLMs generate sloppy code over time, requiring formal analysis and multi-model reviews for quality assurance.
• An Empirical Study of LLM-Generated Specifications for VeriFast — LLMs preserve functional behavior in VeriFast specifications but achieve modest verification success due to domain-specific knowledge gaps.
• The Danger of Vibe-Coding — LLMs are powerful code completion tools, but their next-token prediction mechanism means they lack true understanding, making "vibe coding" dangerous.
• DSLs Enable Reliable Use of LLMs — DSLs provide LLMs with constrained syntax and validation, making code generation reliable and maintainable.
• MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation — MineValiCoder improves LLM-based code generation reliability by mutually validating test case and code quality.
• Harnessing Code Agents for Automatic Software Verification — A general LLM code agent with a verification harness can fully automate formal software proof generation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.