SciDef: Automating Definition Extraction from Academic Literature with Large Language Models

2026-02-05 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, medium

Summary

SciDef is an LLM-based pipeline designed to automate the extraction of definitions from academic literature, addressing the growing challenge of identifying relevant definitions amidst an increasing volume of publications. The system was evaluated using two novel datasets, DefExtra and DefSim, which contain human-extracted definitions and definition-pair similarities, respectively. Testing 16 language models with various prompting strategies, the researchers found that multi-step and DSPy-optimized prompting significantly improved extraction performance. An NLI-based method was identified as the most reliable metric for evaluating extraction quality. SciDef successfully extracted 86.4% of definitions from the test set, indicating strong capability, but also highlighted a tendency for models to over-generate definitions, suggesting future work should focus on relevance identification.

Key takeaway

For AI Scientists and Research Scientists focused on knowledge graph construction or literature review automation, SciDef demonstrates that LLMs can effectively extract definitions from academic texts. You should explore multi-step and DSPy-optimized prompting techniques to improve extraction accuracy in your own LLM-based systems. Be aware that current models may over-generate definitions, necessitating further work on relevance filtering to refine your extracted knowledge.

Key insights

SciDef automates definition extraction from scientific papers using LLMs, achieving 86.4% accuracy with optimized prompting.

Principles

• Multi-step and DSPy prompting enhance LLM extraction.
• NLI-based metrics reliably evaluate definition extraction.

Method

SciDef employs an LLM-based pipeline, testing 16 language models with multi-step and DSPy-optimized prompting strategies, and evaluates extraction using an NLI-based method on DefExtra and DefSim datasets.

In practice

• Use multi-step prompting for better LLM extraction.
• Consider NLI for evaluating definition extraction.

Topics

• Definition Extraction
• Large Language Models
• Academic Literature
• Prompting Strategies
• Natural Language Inference

Code references

• Media-Bias-Group/SciDef

Best for: AI Scientist, Research Scientist, AI Researcher, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Extracting Problem and Method Sentence from Scientific Papers: A Context-enhanced Transformer Using Formulaic Expression Desensitization — Formulaic expression desensitization and context-enhanced transformers improve problem/method sentence extraction from scientific papers, outperforming baselines.
• Extracting Article-Level Legal Dependencies from Swiss Federal Law using LLMs — LLMs can automate the extraction of legal dependencies from complex federal law articles.
• An NLP-Driven Framework for Curriculum-Labor Market Alignment: Schema-Constrained LLM Extraction, ESCO-Anchored Semantic Matching, and Multi-Dimensional Gap Quantification — An NLP framework uses LLMs and semantic matching to quantify curriculum-labor market competency gaps with high reliability.
• yifanfeng97 / Hyper-Extract — Hyper-Extract transforms unstructured text into diverse, strongly-typed knowledge structures using LLM-powered extraction engines and templates.
• Few-Shot Biomedical Relation Extraction with Large Language Models: A Viable Alternative to Supervised Learning? — LLMs offer a viable few-shot alternative for biomedical relation extraction, outperforming prior few-shot methods, especially for rare relations.
• To Describe or Not to Describe? Benchmarking Database Representations for Schema Linking in Text-to-SQL — LLM-generated natural language descriptions enhance Text-to-SQL schema linking more effectively than DDL/DML.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.