Pre-Training on Software Engineering Texts: Effects on Domain Adaptation and General-Language Understanding

2025-03-31 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

This study investigates the effectiveness of adapting Language Models (LMs) for Software Engineering (SE) textual artifacts, such as issues and commit messages, by comparing Continual Pre-Training (CPT) against Pre-Training from Scratch (PTS). Researchers utilized a new 18.5B-token SE corpus, compiled from GitHub, Stack Overflow, Jira, and arXiv, and evaluated LMs ranging from 108M to 7.2B parameters. Experiments were conducted under controlled constant-token (3.3B tokens) and compute-matched (\$6.01\times e^{18}$ FLOPS) budgets. The findings indicate that reusing an existing LM via CPT generally outperforms training a domain-native LM from scratch. CPT yielded small, often inconclusive, domain adaptation gains (measured by SELU) while largely preserving general-language understanding (measured by SuperGLUE). Conversely, PTS incurred significant, usually decisive, penalties on both domain adaptation and general-language understanding, becoming competitive only for smaller LMs with token-rich budgets.

Key takeaway

For Machine Learning Engineers adapting LMs for software engineering textual artifacts, you should prioritize continual pre-training (CPT) over training from scratch (PTS). CPT offers modest domain-specific gains while largely retaining general-language understanding, making it a low-risk refinement. Reserve PTS only when no suitable pre-trained checkpoint exists. Always consider both token and compute budgets, and evaluate performance using both SE-specific and general-language benchmarks to avoid misleading conclusions.

Key insights

Continual pre-training on SE text offers modest domain gains with minimal general-language impact, outperforming training from scratch.

Principles

• CPT generally outperforms PTS for SE text adaptation.
• CPT offers modest domain gains, preserving general-language understanding.
• PTS incurs significant penalties on domain and general understanding.

Method

The study compared Continual Pre-Training (CPT) and Pre-Training from Scratch (PTS) on a new 18.5B-token SE corpus (GitHub, Stack Overflow, Jira, arXiv). Experiments controlled for constant-token (3.3B) and compute-matched (\$6.01\times e^{18}$ FLOPS) budgets, evaluating LMs (108M-7.2B params) on SELU and SuperGLUE benchmarks.

In practice

• Default to existing LMs for SE text tasks.
• Consider CPT for weak baselines or code-focused LMs.
• Evaluate on both SE-specific and general-language benchmarks.

Topics

• Language Model Pre-training
• Software Engineering NLP
• Domain Adaptation
• Continual Pre-Training
• Pre-Training from Scratch
• SELU Benchmark
• SuperGLUE Benchmark

Code references

• huggingface/accelerate

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA — Domain adaptation strategies for medical LLMs show trade-offs between CPT, SFT, and their combination, varying by QA task type.
• Regret Pre-training: Bridging Prior and Posterior Views for Enhanced Knowledge Grounding — Regret Pre-training enhances causal language models by leveraging future context through a teacher-student framework, improving performance without adding parameters.
• On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance — LLM annotation reliability hinges on definition alignment, not just text memorization, as prompt-based corrections are largely ineffective.
• A family of large language models for materials research with insights into model adaptability in continued pretraining — Domain-adapted LLMs significantly enhance materials research, but overtraining can hinder further specialization.
• Citation-Aware Continual Pre-Training for Biomedical Language Models — Incorporating citation graph data into language model pre-training significantly improves biomedical QA accuracy.
• ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning — ShadowPEFT centralizes LLM adaptation via a shared, stateful shadow network, offering modularity and competitive performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.