Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development

2026-07-24 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, extended

Summary

Transformer-Assisted LLM-Based Source Code Summarisation is a new method designed to improve the quality of natural language summaries for source code, crucial for secure software development. This approach combines task-specific Transformer models with Large Language Models (LLMs) by using Transformer-generated summaries within LLM prompt engineering. The research evaluated four LLMs, including Llama 3.1, Deepseek Coder 1.5, and CodeLlama, across various one-shot and zero-shot prompting strategies. The "Explicit One-shot" method, which explicitly provides a Transformer-generated example summary, demonstrated the most significant improvements, yielding a 7.8% increase in Bleu-4 and a 5% increase in BertScore on CodeLlama. The study also found that constraining LLM output length to approximately 9 words enhanced performance and that code-aware LLMs generally surpassed general-purpose LLMs for this task, though Deepseek Coder 1.5 was an exception. The Funcom dataset, updated with a new tokenization method, was used for evaluation.

Key takeaway

For NLP Engineers or Software Engineers tasked with automating source code documentation, you should adopt the "Transformer-Assisted LLM-Based Source Code Summarisation" method. By explicitly providing a Transformer-generated example summary in your LLM prompts, you can achieve significant quality improvements, as demonstrated by a 7.8% Bleu-4 and 5% BertScore increase on CodeLlama. Furthermore, constrain your LLM's output length to around 9 words and prioritize code-aware LLMs for better performance, while being mindful of potential remote code execution risks with certain models.

Key insights

Combining Transformer-generated examples in LLM prompts significantly enhances source code summarization quality.

Principles

• NLG metrics reward lexical overlap, not always semantic quality.
• Code-aware LLMs can excel in Neural Source Code Summarisation (NSCS).
• Explicitly providing examples in prompts boosts LLM performance.

Method

The method, "Transformer-Assisted LLM-Based Source Code Summarisation," involves one-shot prompting LLMs with an explicitly provided example summary generated by a smaller, task-specific Transformer model like CodeSumBart. This example is integrated into the prompt's context.

In practice

• Implement "Explicit One-shot" prompting for NSCS.
• Constrain LLM summary length to ~9 words.
• Select code-aware LLMs for source code tasks.

Topics

• Neural Source Code Summarisation
• Large Language Models
• Prompt Engineering
• Transformer Models
• Secure Software Development
• CodeLlama

Code references

• phillijm/TransformerAssistedLLMCodeSum

Best for: AI Engineer, Machine Learning Engineer, Research Scientist, AI Scientist, NLP Engineer, Software Engineer

Related on AIssential

• Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development — Integrating Transformer-generated summaries into LLM prompts enhances code summarization quality, addressing limitations of standalone methods.
• Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality — LLM code generation quality is biased by prompt language, with English not always superior.
• DSLs Enable Reliable Use of LLMs — DSLs provide LLMs with constrained syntax and validation, making code generation reliable and maintainable.
• Moving To Substack — The Transformer architecture is fundamental to modern generative AI, enabling advanced language model capabilities.
• Label-Aware Pseudo-Training Sample Generation for Text Classification — A new data augmentation method uses LLMs to subtly alter sentences, improving NLP performance with limited data.
• Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation — LLMs' verbalized confidence in translation outputs can be as reliable as internal signals, despite no correlation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.