Large Language Models Have Unreliable Understanding of Software Engineering Terminology

2026-07-08 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, extended

Summary

A study evaluated six Large Language Models (LLMs)—Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.2, GPT-5 Nano, and Gemini 2.5 Flash—on their understanding of standardized software engineering (SE) terminology. Researchers used the ISO/IEC/IEEE 24765:2017 Systems and Software Engineering — Vocabulary, comprising 5,381 definitions, to create a dataset of 4,618 true and systematically falsified definitions. Falsifications were either structural, removing critical information, or semantic, substituting key terms. The findings revealed that while most LLMs detected falsified definitions with high accuracy (True Negative Rate 89.6%-96.2%), they also incorrectly rejected a significant portion of correct definitions (True Positive Rate 16.3%-74.6%). This indicates a systematic "rejection bias" rather than genuine discriminative understanding, with explicit reasoning often failing to improve performance. Opus 4.6-R was an outlier, showing an "acceptance bias" by frequently recognizing flaws but still accepting definitions.

Key takeaway

For Software Engineers or AI Scientists relying on LLMs for tasks requiring precise software engineering terminology, you should explicitly define critical terms within your prompts or context files. Do not assume LLMs possess expert-level, native understanding of SE vocabulary, as they often exhibit a "rejection bias" and struggle with subtle omissions. This proactive approach mitigates risks of miscommunication and faulty outputs stemming from the LLM's superficial semantic understanding.

Key insights

LLMs exhibit a "rejection bias" when judging SE terminology, often failing to accept correct definitions despite high falsification detection.

Principles

• LLMs prioritize syntactic co-occurrence over deep semantic relationships.
• Reasoning tokens do not guarantee improved accuracy in NLU tasks.
• Structural incompleteness is harder for LLMs to detect than semantic shifts.

Method

The study systematically falsified ISO 24765:2017 SE definitions using structural (removing phrases) and semantic (replacing terms via GloVe embeddings) methods. LLMs were prompted to classify definitions as correct/incorrect with reasoning.

In practice

• Explicitly define critical SE terms for LLMs in prompts.
• Do not rely on LLMs for implicit data annotation.
• Verify LLM outputs for terminology accuracy.

Topics

• Large Language Models
• Software Engineering Terminology
• Natural Language Understanding
• ISO 24765:2017
• LLM Bias
• Prompt Engineering

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• The 1.6 Billion People Large Language Models Still Can’t Understand — Large language models fail profoundly in many non-English languages, going beyond mere performance degradation to confident incorrectness.
• Code, Chaos, and the Invisible Logic of LLMs — LLMs predict words probabilistically, not with human intention, leading to potential misunderstandings in code generation.
• What is an LLM? Tokens, Embeddings, and the Big Picture — LLMs operate on subword tokens, not characters, making tokenization a core determinant of their capabilities and predictable failure points.
• From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text — Evaluating LLMs for legal text requires both quantitative benchmarks and qualitative error analysis to reveal true capabilities.
• 4 Lines You Should Include in Your Claude Skill — LLMs can be "confidently wrong" without explicit prompt constraints on context, significance, and confidence.
• True Positive Weekly #166 — The issue highlights diverse AI advancements, from genomic challenges to novel LLM behaviors and practical tools.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.