Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark

2020-10-13 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, extended

Summary

A new multi-domain dataset and benchmark, "Scientific Code Search at Scale," addresses the challenge of discovering relevant scientific software among over 600 million GitHub repositories. This initiative introduces a curated corpus of 5,264 high-quality, domain-classified scientific repositories spanning five NASA Science Mission Directorate divisions. It includes two novel information retrieval benchmarks: a repository search benchmark with 219 expert-curated queries and a large-scale code snippet retrieval benchmark containing 117,950 code snippets and 119,720 queries across seven programming languages. Baseline evaluations reveal significant performance variation across scientific domains and retrieval approaches, with context enrichment consistently improving repository search. Qwen3-Embedding-0.6B achieved the strongest overall performance for code snippet retrieval, outperforming domain-specific models. All datasets and benchmarks are publicly released on HuggingFace.

Key takeaway

For AI Scientists or Machine Learning Engineers developing code search solutions for scientific domains, you should prioritize domain-specific context enrichment and robust handling of varied documentation practices. Recognize that general LLM embeddings like Qwen3-Embedding-0.6B excel at code snippet retrieval, but repository search benefits from scientific-text pretraining. Focus on agentic retrieval systems for challenging identifier-based queries, which current models struggle with significantly.

Key insights

Scientific code discovery requires specialized benchmarks and context enrichment due to domain-specific vocabulary and varied documentation practices.

Principles

• Scientific code search differs significantly from general software engineering tasks.
• Repository documentation quality and inline docstring coverage are distinct factors.
• Context enrichment via external links measurably improves repository retrieval.

Method

Collect scientific repositories from multiple authoritative sources. Apply LLM-based classification and quality filtering. Enrich repository context by cleaning READMEs and crawling external high-signal links.

In practice

• Use LLMs for domain classification and boilerplate removal in documentation.
• Augment sparse repository READMEs with crawled external scientific context.
• Design benchmarks with expert-curated, multi-relevance queries reflecting real scientific needs.

Topics

• Scientific Code Search
• Information Retrieval
• Benchmark Datasets
• NASA Science
• Code Snippet Retrieval
• Large Language Models
• Open Science

Code references

• NASA-IMPACT/github-code-discovery
• unclecode/crawl4ai
• NASA-IMPACT/nasa-science-repo-benchmark

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• TerraMARS: A Domain-Adapted Small-Language-Model Pipeline for Mars Terraforming Literature — Domain-adapted small language models can extract structured scientific data from specialized literature using QLoRA fine-tuning.
• 17 AI Models Tested on REAL Scientific Research — Advanced AI models currently demonstrate limited capability in independent scientific discovery, even with curated data and extensive tools.
• DeepWeb-Bench: A Deep Research Benchmark Demanding Massive Cross-Source Evidence and Long-Horizon Derivation — DeepWeb-Bench challenges frontier LLMs on complex web research tasks requiring extensive evidence and multi-step reasoning.
• On Accelerating Grounded Code Development for Research — Lexical search and structured workflows can effectively ground coding agents in dynamic, domain-specific research.
• CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval — Multimodal code retrieval unifies natural language, code, and images for enhanced code discovery and generation.
• Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning — High-quality domain data and supervised fine-tuning are crucial for reliable LLM-generated Solidity smart contracts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.