Evaluating Semantic and Quality-Aware Retrieval for Source Code Repositories

2026-07-13 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

A study evaluated a prototype retrieval system for source-code repositories, combining function-level fragmentation, OpenAI text-embedding-3-small embeddings, ChromaDB vector storage, and LLM-derived quality metadata. The system supports semantic, quality-filtered, hybrid, and automatic routing retrieval modes. Using a 10% sample of an educational C-code corpus (56 programmer identifiers, 847 files, 3,839 fragments), the evaluation involved 15 manually judged queries. Semantic retrieval demonstrated the strongest aggregate performance, achieving nDCG@5 of 0.820, Success@5 of 0.800, and MRR of 0.644. The automatic router accurately selected the expected mode for all 15 queries but introduced a latency of 1.910 seconds, compared to under 0.2 seconds for direct modes. LLM-derived quality scores aligned with manual assessments within one point for 9 of 12 fragments, though errors occurred with trivial functions. Quality metadata proved most effective for explicitly quality-oriented queries.

Key takeaway

For Machine Learning Engineers building code search systems, prioritize pure semantic retrieval as a strong default for general queries. If you integrate LLM-derived quality metadata, be aware it can distort topical relevance for non-quality-specific searches. Use quality-aware modes primarily for explicitly quality-oriented questions to guide exploratory inspection, but always validate LLM-generated scores, especially for trivial code.

Key insights

Semantic retrieval excels for general code search, while quality metadata enhances quality-specific queries, but can distort topical relevance.

Principles

• LLM-derived quality metadata can provide distinct views of code.
• Quality re-ranking can reduce topical relevance for general queries.
• Retrieval systems are bounded by corpus coverage.

Method

The prototype indexes function-level code fragments, generates LLM-derived quality scores, creates OpenAI text-embedding-3-small embeddings, and stores them in ChromaDB for semantic, quality-filtered, hybrid, or automatic retrieval.

In practice

• Use function-level fragmentation for semantic coherence.
• Implement LLM-as-a-judge for quality scoring at indexing time.
• Group retrieved fragments by programmer identifier for review.

Topics

• Semantic Code Search
• Retrieval-Augmented Generation
• LLM-derived Quality Metadata
• ChromaDB
• Code Embeddings
• C-code Repositories

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Faithful or Findable? Evaluating LLM-Generated Metadata for RDF Dataset Search — LLM-generated metadata boosts dataset search but unconstrained methods risk low faithfulness.
• 40 RAG Interview Questions and Answers — RAG enhances LLM accuracy and reduces hallucinations by providing real-time, verifiable external knowledge.
• Balancing Usefulness and Naturalness: An LLM-based Curation Pipeline for Code Review Comments — Curating LLM training data for code review comments by balancing quality and natural diversity significantly improves model performance.
• MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval — Topic metadata, selected and abstracted, can efficiently guide RAG retrieval over coarse chunks, improving precision and latency.
• Combining Retrieval-Augmented Text Generation with LLMs for Reading Content Recommendations — RAG significantly enhances LLM-generated personalized reading content, improving relevance and groundedness.
• Context-as-a-Service: Surfacing Cross-File Dependency Chains for LLM-Generated Developer Documentation — CaaS enhances LLM documentation agents by surfacing non-obvious cross-file dependencies, improving accuracy and efficiency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.