A family of large language models for materials research with insights into model adaptability in continued pretraining

2026-02-27 · Source: Nature Machine Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, long

Summary

LLaMat, a new family of large language models, has been developed for materials science research through continued pretraining of LLaMA models. Researchers used 30 billion tokens from approximately 4 million materials science publications and crystallographic data. The models were further adapted for a materials copilot by instruction and task fine-tuning on 175,000 materials science question-answering pairs. LLaMat consistently outperforms commercial LLMs like Claude, GPT, and Gemini across 42 tasks, including natural language processing, structured information extraction, and crystal generation, while retaining general linguistic capabilities. The study also identified "adaptation rigidity" in extensively pretrained LLMs such as LLaMA-3, where overtrained models show increasing resistance to domain-specific adaptation.

Key takeaway

For AI Scientists developing specialized scientific AI systems, you should prioritize domain-specific continued pretraining and fine-tuning, as demonstrated by LLaMat's superior performance. However, be aware of the identified "adaptation rigidity" in highly pretrained base models like LLaMA-3, which suggests that selecting a less extensively pretrained foundation model might offer greater flexibility for deep domain adaptation.

Key insights

Domain-adapted LLMs significantly enhance materials research, but overtraining can hinder further specialization.

Principles

• Domain adaptation improves LLM performance.
• Extensively pretrained LLMs exhibit "adaptation rigidity".

Method

LLaMA models were continually pretrained on 30 billion materials science tokens, then instruction and task fine-tuned on 175,000 Q&A pairs for materials research.

In practice

• Use LLaMat for materials science NLP tasks.
• Consider adaptation rigidity in LLM selection.

Topics

• Materials Large Language Models
• Domain Adaptation
• Continued Pretraining
• Materials Discovery
• Adaptation Rigidity

Code references

• M3RG-IITD/llamat
• facebookresearch/crystal-text-llm
• M3RG-IITD/llamat

Best for: AI Scientist, Research Scientist, AI Researcher, Machine Learning Engineer, Data Scientist

Related on AIssential

• IEEE Transactions on Artificial Intelligence, Volume 7, Issue 2, February 2026 — The volume showcases diverse AI advancements, from LLM applications in autonomous driving to specialized models for protein language processing.
• A multimodal large language model for materials science — MatterChat unifies material structure and text using a modular LLM, improving property prediction and scientific reasoning in materials science.
• Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation — RoPE-Perturbed Self-Distillation improves LLM long-context understanding by reducing reliance on absolute positional encoding.
• Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning — DOMINO inductively synthesizes domain-specific data for LLMs by learning minimal representations from reference examples, bypassing explicit domain descriptions.
• Mutation Without Variation: Convergence Dynamics in LLM-Driven Program Evolution — LLM-driven program mutation inherently converges to limited structural forms, hindering open-ended exploration.
• Revisiting LLM Adaptation for 3D CT Report Generation: A Study of Scaling and Diagnostic Priors — RAD3D-Prefix enables efficient 3D CT report generation by adapting frozen LLMs with diagnostic priors.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Nature Machine Intelligence.